Jul 17 18:09:48.179457 ip-10-0-137-221 systemd[1]: Starting Kubernetes Kubelet... Jul 17 18:09:49.849003 ip-10-0-137-221 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:09:49.849003 ip-10-0-137-221 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 18:09:49.849003 ip-10-0-137-221 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:09:49.849003 ip-10-0-137-221 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 18:09:49.849003 ip-10-0-137-221 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:09:49.850081 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.849904 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 18:09:49.854844 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854828 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:49.854844 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854844 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854848 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854851 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854854 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854857 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854861 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854863 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854866 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854870 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854872 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854875 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854877 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854880 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854883 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854885 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854888 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854891 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854893 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854895 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854898 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:49.854907 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854900 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854903 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854908 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854912 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854937 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854967 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854974 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854977 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854980 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854984 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854987 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854989 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854992 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854995 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.854999 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855003 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855006 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855008 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855011 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:49.855404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855013 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855016 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855019 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855022 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855025 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855028 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855031 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855034 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855036 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855039 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855041 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855044 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855047 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855050 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855052 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855055 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855058 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855061 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855066 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:49.855918 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855069 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855072 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855077 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855080 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855083 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855085 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855088 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855091 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855095 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855099 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855102 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855105 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855111 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855113 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855116 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855119 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855121 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855125 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855129 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:49.856388 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855132 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855135 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855137 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855140 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855143 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855145 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855148 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855150 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855153 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855156 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855158 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855161 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855165 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855168 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855171 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855174 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855176 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855179 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855182 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855184 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:49.856853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855187 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855946 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855969 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855974 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855977 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855980 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855983 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855985 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855988 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855990 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855993 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855996 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.855999 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856002 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856004 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856007 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856009 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856012 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856014 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:49.857344 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856017 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856019 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856022 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856025 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856028 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856031 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856034 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856039 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856042 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856046 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856049 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856052 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856055 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856058 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856061 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856065 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856067 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856070 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856073 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856076 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:49.857810 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856078 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856082 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856086 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856090 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856092 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856095 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856098 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856101 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856104 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856107 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856110 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856112 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856114 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856117 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856120 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856123 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856125 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856128 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856131 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:49.858310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856134 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856137 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856140 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856143 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856145 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856148 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856151 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856153 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856156 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856158 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856161 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856163 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856166 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856169 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856171 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856174 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856176 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856178 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856181 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:49.858775 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856183 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856186 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856188 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856191 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856193 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856195 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856198 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856200 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856202 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856205 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856208 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856211 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856215 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856217 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856220 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856224 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856226 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856229 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856231 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856233 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:49.859256 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856236 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856239 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856241 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856606 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856611 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856614 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856616 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856618 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856621 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856623 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856626 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856628 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856631 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856633 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856636 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856638 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856641 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856643 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856645 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856648 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856650 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:49.859761 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856653 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856655 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856659 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856662 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856664 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856667 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856670 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856673 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856675 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856678 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856680 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856683 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856686 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856688 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856691 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856693 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856695 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856698 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856700 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856703 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:49.860288 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856706 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856710 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856713 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856717 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856720 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856723 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856726 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856728 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856731 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856733 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856736 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856738 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856741 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856743 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856749 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856752 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856754 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856756 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856759 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:49.860779 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856762 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856765 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856768 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856770 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856773 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856775 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856778 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856780 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856783 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856785 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856788 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856790 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856793 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856796 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856799 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856801 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856804 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856806 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856809 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856811 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:49.861277 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856813 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856816 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856820 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856824 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856827 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856830 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856833 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856837 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856839 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856842 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856844 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856847 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856849 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856853 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856855 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856858 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856860 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856863 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856865 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:49.861750 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856868 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856870 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.856873 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856944 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856965 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856973 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856979 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856983 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856987 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856991 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856995 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.856998 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857001 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857005 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857008 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857011 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857014 2570 flags.go:64] FLAG: --cgroup-root="" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857017 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857020 2570 flags.go:64] FLAG: --client-ca-file="" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857023 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857027 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857032 2570 flags.go:64] FLAG: --cluster-domain="" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857034 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 18:09:49.862221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857037 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857041 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857044 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857047 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857054 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857057 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857061 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857064 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857067 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857069 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857073 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857076 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857080 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857083 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857086 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857089 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857093 2570 flags.go:64] FLAG: --enable-server="true" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857096 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857100 2570 flags.go:64] FLAG: --event-burst="100" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857104 2570 flags.go:64] FLAG: --event-qps="50" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857106 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857110 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857113 2570 flags.go:64] FLAG: --eviction-hard="" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857117 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 18:09:49.862809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857119 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857123 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857126 2570 flags.go:64] FLAG: --eviction-soft="" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857128 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857133 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857138 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857141 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857144 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857147 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857150 2570 flags.go:64] FLAG: --feature-gates="" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857154 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857157 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857160 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857163 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857166 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857169 2570 flags.go:64] FLAG: --help="false" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857172 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-137-221.ec2.internal" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857175 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857178 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857181 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857185 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857188 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857191 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 18:09:49.863419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857193 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857196 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857199 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857202 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857205 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857207 2570 flags.go:64] FLAG: --kube-reserved="" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857210 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857213 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857216 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857218 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857221 2570 flags.go:64] FLAG: --lock-file="" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857224 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857227 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857231 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857238 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857240 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857243 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857246 2570 flags.go:64] FLAG: --logging-format="text" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857249 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857252 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857255 2570 flags.go:64] FLAG: --manifest-url="" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857258 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857262 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857265 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857269 2570 flags.go:64] FLAG: --max-pods="110" Jul 17 18:09:49.864291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857273 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857276 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857279 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857282 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857285 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857288 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857291 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857299 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857302 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857305 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857308 2570 flags.go:64] FLAG: --pod-cidr="" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857310 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857316 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857319 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857322 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857325 2570 flags.go:64] FLAG: --port="10250" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857328 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857331 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a9b570d74d270e31" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857334 2570 flags.go:64] FLAG: --qos-reserved="" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857337 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857344 2570 flags.go:64] FLAG: --register-node="true" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857347 2570 flags.go:64] FLAG: --register-with-taints="" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857352 2570 flags.go:64] FLAG: --registry-burst="10" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857356 2570 flags.go:64] FLAG: --registry-qps="5" Jul 17 18:09:49.865159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857359 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857362 2570 flags.go:64] FLAG: --reserved-memory="" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857366 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857369 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857372 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857375 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857378 2570 flags.go:64] FLAG: --runonce="false" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857381 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857384 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857387 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857390 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857392 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857395 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857398 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857401 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857404 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857407 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857410 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857413 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857416 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857419 2570 flags.go:64] FLAG: --system-cgroups="" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857421 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857427 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857430 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857432 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 18:09:49.865745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857437 2570 flags.go:64] FLAG: --tls-min-version="" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857440 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857442 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857446 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857449 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857453 2570 flags.go:64] FLAG: --v="2" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857462 2570 flags.go:64] FLAG: --version="false" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857466 2570 flags.go:64] FLAG: --vmodule="" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857469 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.857473 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857586 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857589 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857592 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857595 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857598 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857602 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857605 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857607 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857609 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857612 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857615 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:49.866460 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857619 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857622 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857625 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857629 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857631 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857634 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857636 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857639 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857641 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857644 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857646 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857649 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857652 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857654 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857658 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857660 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857664 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857667 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857669 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857672 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:49.867105 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857674 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857677 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857679 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857682 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857684 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857687 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857689 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857692 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857694 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857697 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857699 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857702 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857704 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857706 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857709 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857711 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857714 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857716 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857718 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857721 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:49.867619 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857724 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857726 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857729 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857731 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857733 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857736 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857740 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857742 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857746 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857749 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857751 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857754 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857757 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857759 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857761 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857764 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857767 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857770 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857772 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857775 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:49.868126 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857778 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857780 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857783 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857786 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857788 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857791 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857795 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857798 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857801 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857804 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857806 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857809 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857811 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857814 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857817 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857819 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857822 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857825 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857828 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:49.868602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857831 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857834 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857836 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857839 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857841 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857844 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857846 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857848 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.857852 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.858357 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.866305 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.866324 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.867033 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.867041 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 18:09:49.869069 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867105 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867110 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867113 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867117 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867120 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867122 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867125 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867128 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867130 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867133 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867136 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867139 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867141 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867144 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867146 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867149 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867152 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867155 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867157 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867160 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:49.869427 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867162 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867165 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867167 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867170 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867172 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867175 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867177 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867180 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867182 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867185 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867188 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867191 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867193 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867196 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867199 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867201 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867204 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867207 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867209 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867212 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:49.869912 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867214 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867217 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867219 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867222 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867225 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867227 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867231 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867234 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867236 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867240 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867244 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867247 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867250 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867253 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867255 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867258 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867261 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867263 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867266 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867269 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:49.870413 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867271 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867274 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867277 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867280 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867283 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867285 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867288 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867291 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867293 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867296 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867298 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867301 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867303 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867306 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867308 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867311 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867313 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867316 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867318 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:49.870941 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867322 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867324 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867327 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867331 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867334 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867337 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867340 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867344 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867347 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867350 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867353 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867355 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867358 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867360 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867363 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867366 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867369 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867371 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867374 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:49.871504 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867377 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.867382 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867515 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867520 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867523 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867526 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867529 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867533 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867536 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867540 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867542 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867545 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867548 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867551 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867554 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:49.871974 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867556 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867559 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867561 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867563 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867567 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867569 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867572 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867574 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867576 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867579 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867581 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867584 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867586 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867589 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867592 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867594 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867597 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867599 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867601 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867604 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:49.872354 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867606 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867609 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867611 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867613 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867616 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867618 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867622 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867626 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867629 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867631 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867635 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867638 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867642 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867645 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867648 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867650 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867653 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867656 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:49.872834 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867658 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867661 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867663 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867666 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867668 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867671 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867674 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867676 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867679 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867681 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867684 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867687 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867689 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867692 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867694 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867697 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867699 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867701 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867704 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867706 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:49.873275 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867709 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867711 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867714 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867716 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867719 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867722 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867724 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867727 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867729 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867731 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867734 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867737 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867739 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867741 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867744 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867746 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867748 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867751 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867753 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867756 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:49.873770 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867758 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867761 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867764 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867766 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867769 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867771 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867774 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:49.867776 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.867782 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.867893 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.869939 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.871082 2570 server.go:1019] "Starting client certificate rotation" Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.871175 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.871213 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:09:49.874265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.873395 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:09:49.876000 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.875983 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:09:49.885399 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.885380 2570 log.go:25] "Validated CRI v1 runtime API" Jul 17 18:09:49.887356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.887343 2570 log.go:25] "Validated CRI v1 image API" Jul 17 18:09:49.888867 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.888854 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 18:09:49.891458 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.891437 2570 fs.go:135] Filesystem UUIDs: map[46f00773-caf7-42e2-ba76-6ab8a256b196:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b40e054e-a485-4c49-a20f-01cc67baa5bc:/dev/nvme0n1p3] Jul 17 18:09:49.891533 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.891459 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 18:09:49.894753 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.894733 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:09:49.897966 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.897850 2570 manager.go:216] Machine: {Timestamp:2026-07-17 18:09:49.896672085 +0000 UTC m=+0.311410103 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110099 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec205b97202a7670505baadc237499d2 SystemUUID:ec205b97-202a-7670-505b-aadc237499d2 BootID:fd5e080f-f9c9-4e75-b38d-bc67b9b754b9 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:25:f6:14:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:25:f6:14:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:64:f7:a5:2f:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 18:09:49.897966 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.897949 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 18:09:49.898068 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.898047 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 18:09:49.900656 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.900627 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 18:09:49.900819 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.900659 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 18:09:49.901284 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.901273 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 18:09:49.901347 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.901291 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 18:09:49.901347 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.901301 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:09:49.901901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.901890 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:09:49.903404 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.903391 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 18:09:49.906196 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906182 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:09:49.906332 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906320 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 18:09:49.906398 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906359 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 17 18:09:49.906398 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906374 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 18:09:49.906398 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906389 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 18:09:49.906516 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906405 2570 kubelet.go:398] "Adding apiserver pod source" Jul 17 18:09:49.906516 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906417 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 18:09:49.906811 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906797 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:09:49.906871 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.906820 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:09:49.908843 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.908829 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 18:09:49.910197 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910184 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 18:09:49.910248 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910215 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 18:09:49.910765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910754 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 18:09:49.910801 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910775 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 18:09:49.910801 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910784 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 18:09:49.910801 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910789 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 18:09:49.910801 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910798 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 18:09:49.910908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910804 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 18:09:49.910908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910857 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 18:09:49.910908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910863 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 18:09:49.910908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910869 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 18:09:49.910908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910875 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 18:09:49.910908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910883 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 18:09:49.910908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.910892 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 18:09:49.911681 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.911671 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 18:09:49.911681 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.911680 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 18:09:49.914947 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.914930 2570 server.go:1268] "Started kubelet" Jul 17 18:09:49.915048 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.915022 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 18:09:49.915389 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.915216 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 18:09:49.915389 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.915290 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 18:09:49.915898 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.915866 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 18:09:49.915893 ip-10-0-137-221 systemd[1]: Started Kubernetes Kubelet. Jul 17 18:09:49.916093 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.915933 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 18:09:49.916093 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.916075 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 18:09:49.916525 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.916493 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 18:09:49.917386 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.917374 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 17 18:09:49.919163 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.919142 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rhg75" Jul 17 18:09:49.919456 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.919435 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 18:09:49.922988 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.921353 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-221.ec2.internal.18c3261e866b4752 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-221.ec2.internal,UID:ip-10-0-137-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-221.ec2.internal,},FirstTimestamp:2026-07-17 18:09:49.914900306 +0000 UTC m=+0.329638324,LastTimestamp:2026-07-17 18:09:49.914900306 +0000 UTC m=+0.329638324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-221.ec2.internal,}" Jul 17 18:09:49.925347 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.925328 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 18:09:49.925347 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.925340 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 18:09:49.925500 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.925400 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:49.925707 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.925689 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rhg75" Jul 17 18:09:49.925938 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.925917 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 18:09:49.925938 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.925916 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 18:09:49.926078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.925945 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 18:09:49.926078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926068 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 17 18:09:49.926078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926076 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 17 18:09:49.926212 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926113 2570 factory.go:55] Registering systemd factory Jul 17 18:09:49.926212 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926127 2570 factory.go:223] Registration of the systemd container factory successfully Jul 17 18:09:49.926366 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926350 2570 factory.go:153] Registering CRI-O factory Jul 17 18:09:49.926366 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926366 2570 factory.go:223] Registration of the crio container factory successfully Jul 17 18:09:49.926671 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926654 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 18:09:49.926739 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926690 2570 factory.go:103] Registering Raw factory Jul 17 18:09:49.926739 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.926708 2570 manager.go:1196] Started watching for new ooms in manager Jul 17 18:09:49.929131 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.929096 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 18:09:49.929266 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.929135 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 18:09:49.929627 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.929416 2570 manager.go:349] Starting recovery of all containers Jul 17 18:09:49.939915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.939898 2570 manager.go:354] Recovery completed Jul 17 18:09:49.944343 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.944330 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:49.946132 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.946117 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:49.946186 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.946146 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:49.946186 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.946159 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:49.946682 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.946668 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 18:09:49.946682 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.946680 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 18:09:49.946772 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.946694 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:09:49.946772 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.946741 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:49.947525 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.947510 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:49.947586 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.947535 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:49.947586 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.947544 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:49.947849 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.947838 2570 policy_none.go:49] "None policy: Start" Jul 17 18:09:49.947885 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.947853 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 18:09:49.947885 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.947863 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 18:09:49.948776 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.948765 2570 policy_none.go:47] "Start" Jul 17 18:09:49.987793 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.987585 2570 manager.go:337] "Starting Device Plugin manager" Jul 17 18:09:49.987927 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.987808 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 18:09:49.987927 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.987822 2570 server.go:87] "Starting device plugin registration server" Jul 17 18:09:49.988136 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.988123 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 18:09:49.988181 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.988139 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 18:09:49.988414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.988257 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 18:09:49.988414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.988369 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 18:09:49.988414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:49.988378 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 18:09:49.988974 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.988922 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 18:09:49.989090 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:49.989004 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.064703 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.064668 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 18:09:50.066263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.065827 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 18:09:50.066263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.065869 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 18:09:50.066263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.065898 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 18:09:50.066263 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.065942 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 18:09:50.067974 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.067943 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:50.088626 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.088601 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:50.089500 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.089484 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:50.089567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.089514 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:50.089567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.089524 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:50.089567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.089546 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.097709 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.097687 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.097811 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.097713 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-221.ec2.internal\": node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.112840 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.112816 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.166939 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.166908 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal"] Jul 17 18:09:50.167034 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.167013 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:50.168024 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.168006 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:50.168107 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.168036 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:50.168107 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.168048 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:50.169853 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.169840 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:50.169983 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.169940 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.169983 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.169981 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:50.170494 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.170475 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:50.170585 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.170498 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:50.170585 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.170519 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:50.170585 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.170528 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:50.170585 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.170501 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:50.170585 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.170581 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:50.172330 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.172315 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.172414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.172340 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:50.172981 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.172946 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:50.173068 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.172991 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:50.173068 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.173005 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:50.202727 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.202707 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-221.ec2.internal\" not found" node="ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.206140 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.206121 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-221.ec2.internal\" not found" node="ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.212877 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.212858 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.227333 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.227298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51056fe6463a449526dfa7fd7c89f346-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal\" (UID: \"51056fe6463a449526dfa7fd7c89f346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.227445 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.227341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a98b89d541a53d94896a6b1c12baf188-config\") pod \"kube-apiserver-proxy-ip-10-0-137-221.ec2.internal\" (UID: \"a98b89d541a53d94896a6b1c12baf188\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.227445 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.227359 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51056fe6463a449526dfa7fd7c89f346-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal\" (UID: \"51056fe6463a449526dfa7fd7c89f346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.313647 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.313611 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.328083 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.327994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51056fe6463a449526dfa7fd7c89f346-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal\" (UID: \"51056fe6463a449526dfa7fd7c89f346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.328083 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.328038 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a98b89d541a53d94896a6b1c12baf188-config\") pod \"kube-apiserver-proxy-ip-10-0-137-221.ec2.internal\" (UID: \"a98b89d541a53d94896a6b1c12baf188\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.328083 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.328056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51056fe6463a449526dfa7fd7c89f346-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal\" (UID: \"51056fe6463a449526dfa7fd7c89f346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.328310 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.328103 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a98b89d541a53d94896a6b1c12baf188-config\") pod \"kube-apiserver-proxy-ip-10-0-137-221.ec2.internal\" (UID: \"a98b89d541a53d94896a6b1c12baf188\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.328310 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.328110 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51056fe6463a449526dfa7fd7c89f346-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal\" (UID: \"51056fe6463a449526dfa7fd7c89f346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.328310 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.328156 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51056fe6463a449526dfa7fd7c89f346-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal\" (UID: \"51056fe6463a449526dfa7fd7c89f346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.414436 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.414400 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.504988 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.504935 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.510612 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.510595 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" Jul 17 18:09:50.515228 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.515204 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.615759 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.615676 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.716236 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.716206 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.816892 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.816864 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.871385 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.871306 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 18:09:50.871815 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.871471 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 18:09:50.916969 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:50.916923 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-221.ec2.internal\" not found" Jul 17 18:09:50.930915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.930879 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 18:04:49 +0000 UTC" deadline="2028-01-09 17:57:36.145755096 +0000 UTC" Jul 17 18:09:50.930915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.930903 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12983h47m45.214854482s" Jul 17 18:09:50.973891 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.973862 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:50.985085 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:50.985062 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:51.025895 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.025858 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" Jul 17 18:09:51.037224 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.037200 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:09:51.038095 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.038081 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" Jul 17 18:09:51.046703 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.046682 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:09:51.132698 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:51.132666 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98b89d541a53d94896a6b1c12baf188.slice/crio-3be2d712341c040df728aa6c02d4f58b42c24b80ac64adf14654ab234e09236c WatchSource:0}: Error finding container 3be2d712341c040df728aa6c02d4f58b42c24b80ac64adf14654ab234e09236c: Status 404 returned error can't find the container with id 3be2d712341c040df728aa6c02d4f58b42c24b80ac64adf14654ab234e09236c Jul 17 18:09:51.133269 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:51.133249 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51056fe6463a449526dfa7fd7c89f346.slice/crio-b678633dacff1c3aae55d688d697c9ea30f39fdfd30418d11c40d425c097d9ef WatchSource:0}: Error finding container b678633dacff1c3aae55d688d697c9ea30f39fdfd30418d11c40d425c097d9ef: Status 404 returned error can't find the container with id b678633dacff1c3aae55d688d697c9ea30f39fdfd30418d11c40d425c097d9ef Jul 17 18:09:51.138445 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.138428 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:09:51.405091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.405013 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:51.764334 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.764258 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:51.908045 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.908016 2570 apiserver.go:52] "Watching apiserver" Jul 17 18:09:51.915589 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.915237 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 18:09:51.917749 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.917717 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-5hh2k","openshift-ovn-kubernetes/ovnkube-node-txxzb","kube-system/global-pull-secret-syncer-zv9q4","kube-system/konnectivity-agent-qrr5b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb","openshift-dns/node-resolver-52kz4","openshift-image-registry/node-ca-btwwl","openshift-multus/multus-45tjm","openshift-multus/multus-additional-cni-plugins-24fhs","openshift-multus/network-metrics-daemon-xlrp2","kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal","openshift-cluster-node-tuning-operator/tuned-z6s9b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal","openshift-network-diagnostics/network-check-target-gxfcg"] Jul 17 18:09:51.920293 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.920268 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:51.923356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.922942 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 18:09:51.923356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.923066 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztbhl\"" Jul 17 18:09:51.923356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.923083 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 18:09:51.923356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.923203 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 18:09:51.923598 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.923377 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.925369 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.925348 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 18:09:51.925898 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.925530 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 18:09:51.925898 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.925681 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lld4k\"" Jul 17 18:09:51.926146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.926131 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 18:09:51.926218 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.926153 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 18:09:51.926218 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.926163 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 18:09:51.926345 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.926332 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:51.926446 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:51.926411 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:09:51.928374 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.928165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:51.930180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.930156 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 18:09:51.930456 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.930437 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zx66f\"" Jul 17 18:09:51.930530 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.930500 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 18:09:51.931899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.931876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:51.934179 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.934164 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 18:09:51.934415 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.934397 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 18:09:51.934493 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.934410 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 18:09:51.935133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.935066 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:51.935133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.935089 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lxtgc\"" Jul 17 18:09:51.937719 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.937700 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:51.937719 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.937714 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 18:09:51.937894 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.937791 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jn8lw\"" Jul 17 18:09:51.937993 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.937938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83445c09-b601-4b2a-b503-1d4d7138ee68-konnectivity-ca\") pod \"konnectivity-agent-qrr5b\" (UID: \"83445c09-b601-4b2a-b503-1d4d7138ee68\") " pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:51.938071 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.937999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-cni-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938071 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938025 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-os-release\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938071 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-cni-binary-copy\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-k8s-cni-cncf-io\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-cni-bin\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-kubelet\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938145 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:51.938216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938152 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:09:51.938216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0405c6f-6271-4413-aed6-8bf5750e553e-host\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:51.938216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938188 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0405c6f-6271-4413-aed6-8bf5750e553e-serviceca\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938220 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm7rn\" (UniqueName: \"kubernetes.io/projected/c0405c6f-6271-4413-aed6-8bf5750e553e-kube-api-access-gm7rn\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938153 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-system-cni-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938269 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-netns\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938295 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-cni-multus\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-hostroot\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32593b2d-c97f-4e09-941c-0c2aaa602360-kubelet-config\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32593b2d-c97f-4e09-941c-0c2aaa602360-dbus\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-cnibin\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-multus-certs\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jr4\" (UniqueName: \"kubernetes.io/projected/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-kube-api-access-s9jr4\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83445c09-b601-4b2a-b503-1d4d7138ee68-agent-certs\") pod \"konnectivity-agent-qrr5b\" (UID: \"83445c09-b601-4b2a-b503-1d4d7138ee68\") " pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-socket-dir-parent\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-conf-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-daemon-config\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.938572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.938569 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-etc-kubernetes\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:51.940292 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.940104 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 18:09:51.940292 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.940133 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 18:09:51.940292 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.940288 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:51.940823 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.940806 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g2rkf\"" Jul 17 18:09:51.942430 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.942410 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:09:51.942804 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.942786 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 18:09:51.942991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.942974 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 18:09:51.943057 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.943003 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8qgnd\"" Jul 17 18:09:51.943057 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.943015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:51.945052 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:51.945141 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945101 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 18:09:51.945396 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dlx5g\"" Jul 17 18:09:51.945436 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:51.945111 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:09:51.945510 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945491 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 18:09:51.945510 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945508 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 18:09:51.946038 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945823 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 18:09:51.946038 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945842 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 18:09:51.946038 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.945891 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 18:09:51.947421 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.947407 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:51.947647 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:51.947621 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:09:51.949966 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.949929 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:51.952446 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.952121 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 18:09:51.952446 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.952320 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:09:51.952750 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.952729 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m4jrg\"" Jul 17 18:09:51.956427 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.956403 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-flpwt" Jul 17 18:09:51.963677 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:51.963651 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-flpwt" Jul 17 18:09:52.030232 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.029616 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 18:09:52.039015 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.038979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-cni-bin\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.039177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83445c09-b601-4b2a-b503-1d4d7138ee68-konnectivity-ca\") pod \"konnectivity-agent-qrr5b\" (UID: \"83445c09-b601-4b2a-b503-1d4d7138ee68\") " pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:52.039177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-os-release\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-cni-bin\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-kubelet\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039118 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-modprobe-d\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.039177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039138 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-var-lib-kubelet\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.039177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-etc-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039178 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-os-release\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039183 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-env-overrides\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039223 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-system-cni-dir\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039240 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-cni-bin\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-hostroot\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-sys\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039282 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-kubelet\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5276339c-240f-48ee-afad-f81746946eb8-iptables-alerter-script\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-ovn\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5276339c-240f-48ee-afad-f81746946eb8-host-slash\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-hostroot\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039376 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-run-ovn-kubernetes\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovn-node-metrics-cert\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039443 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-cni-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-device-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039506 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:52.039529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-hosts-file\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039556 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-cni-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039613 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-cni-multus\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-sys-fs\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49b9n\" (UniqueName: \"kubernetes.io/projected/6198f643-e16b-4409-a9de-a353b91e305c-kube-api-access-49b9n\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysctl-conf\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.039696 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039699 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-run\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039720 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-lib-modules\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-os-release\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.039781 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret podName:32593b2d-c97f-4e09-941c-0c2aaa602360 nodeName:}" failed. No retries permitted until 2026-07-17 18:09:52.539745353 +0000 UTC m=+2.954483378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret") pod "global-pull-secret-syncer-zv9q4" (UID: "32593b2d-c97f-4e09-941c-0c2aaa602360") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039794 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-var-lib-cni-multus\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039808 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83445c09-b601-4b2a-b503-1d4d7138ee68-konnectivity-ca\") pod \"konnectivity-agent-qrr5b\" (UID: \"83445c09-b601-4b2a-b503-1d4d7138ee68\") " pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039814 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-systemd\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.040316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6vs\" (UniqueName: \"kubernetes.io/projected/e427f223-46d1-4317-a69e-7882713600cd-kube-api-access-nf6vs\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039926 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwkm\" (UniqueName: \"kubernetes.io/projected/5276339c-240f-48ee-afad-f81746946eb8-kube-api-access-snwkm\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039972 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-kubelet\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.039997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-systemd-units\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-log-socket\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83445c09-b601-4b2a-b503-1d4d7138ee68-agent-certs\") pod \"konnectivity-agent-qrr5b\" (UID: \"83445c09-b601-4b2a-b503-1d4d7138ee68\") " pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-conf-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040189 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-daemon-config\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040214 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-socket-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040282 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-conf-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040437 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-registration-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040476 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-cni-netd\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040504 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040533 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-cni-binary-copy\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040600 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dacf6111-b222-4313-b252-ef561ed2745d-tmp\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.041146 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0405c6f-6271-4413-aed6-8bf5750e553e-serviceca\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040723 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm7rn\" (UniqueName: \"kubernetes.io/projected/c0405c6f-6271-4413-aed6-8bf5750e553e-kube-api-access-gm7rn\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040744 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysconfig\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040793 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-daemon-config\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040914 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz88l\" (UniqueName: \"kubernetes.io/projected/dacf6111-b222-4313-b252-ef561ed2745d-kube-api-access-xz88l\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7hb\" (UniqueName: \"kubernetes.io/projected/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-kube-api-access-wf7hb\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.040994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-slash\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32593b2d-c97f-4e09-941c-0c2aaa602360-kubelet-config\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041066 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-multus-certs\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041085 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-cni-binary-copy\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041097 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32593b2d-c97f-4e09-941c-0c2aaa602360-kubelet-config\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041157 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jr4\" (UniqueName: \"kubernetes.io/projected/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-kube-api-access-s9jr4\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0405c6f-6271-4413-aed6-8bf5750e553e-serviceca\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-tmp-dir\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041280 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-multus-certs\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-cnibin\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.041800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-socket-dir-parent\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041401 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysctl-d\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-var-lib-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041454 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-multus-socket-dir-parent\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041482 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovnkube-script-lib\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-k8s-cni-cncf-io\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041577 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041608 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-k8s-cni-cncf-io\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-systemd\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041704 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-host\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0405c6f-6271-4413-aed6-8bf5750e553e-host\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041739 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-system-cni-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041757 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-netns\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.042541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnp9c\" (UniqueName: \"kubernetes.io/projected/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-kube-api-access-nnp9c\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041807 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-kubernetes\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041832 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32593b2d-c97f-4e09-941c-0c2aaa602360-dbus\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-cnibin\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041873 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-system-cni-dir\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-run-netns\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-host-run-netns\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041927 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0405c6f-6271-4413-aed6-8bf5750e553e-host\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-node-log\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.041986 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovnkube-config\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.042003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32593b2d-c97f-4e09-941c-0c2aaa602360-dbus\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.042012 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lc7q\" (UniqueName: \"kubernetes.io/projected/03f309d6-aee5-4407-bfc2-a3f9b61cd006-kube-api-access-6lc7q\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.042039 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-etc-kubernetes\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.042045 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-cnibin\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.042071 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.042079 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-etc-kubernetes\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.043213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.042096 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dacf6111-b222-4313-b252-ef561ed2745d-etc-tuned\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.044126 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.044107 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83445c09-b601-4b2a-b503-1d4d7138ee68-agent-certs\") pod \"konnectivity-agent-qrr5b\" (UID: \"83445c09-b601-4b2a-b503-1d4d7138ee68\") " pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:52.050880 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.050845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm7rn\" (UniqueName: \"kubernetes.io/projected/c0405c6f-6271-4413-aed6-8bf5750e553e-kube-api-access-gm7rn\") pod \"node-ca-btwwl\" (UID: \"c0405c6f-6271-4413-aed6-8bf5750e553e\") " pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:52.051015 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.050990 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jr4\" (UniqueName: \"kubernetes.io/projected/2d4a95c5-11f5-4b4b-a867-6a11cbd3aded-kube-api-access-s9jr4\") pod \"multus-45tjm\" (UID: \"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded\") " pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.070087 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.070027 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" event={"ID":"51056fe6463a449526dfa7fd7c89f346","Type":"ContainerStarted","Data":"b678633dacff1c3aae55d688d697c9ea30f39fdfd30418d11c40d425c097d9ef"} Jul 17 18:09:52.071356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.071326 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" event={"ID":"a98b89d541a53d94896a6b1c12baf188","Type":"ContainerStarted","Data":"3be2d712341c040df728aa6c02d4f58b42c24b80ac64adf14654ab234e09236c"} Jul 17 18:09:52.142685 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-run-netns\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.142685 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-node-log\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.142893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovnkube-config\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.142893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-run-netns\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.142893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142775 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lc7q\" (UniqueName: \"kubernetes.io/projected/03f309d6-aee5-4407-bfc2-a3f9b61cd006-kube-api-access-6lc7q\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.142893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:52.142893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142834 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-node-log\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.142893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dacf6111-b222-4313-b252-ef561ed2745d-etc-tuned\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.142893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-cni-bin\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-modprobe-d\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-var-lib-kubelet\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-etc-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.142992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-env-overrides\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-system-cni-dir\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-sys\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5276339c-240f-48ee-afad-f81746946eb8-iptables-alerter-script\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-ovn\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143118 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-modprobe-d\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5276339c-240f-48ee-afad-f81746946eb8-host-slash\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143160 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-run-ovn-kubernetes\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-cni-bin\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143184 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovn-node-metrics-cert\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143189 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-system-cni-dir\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-device-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143222 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-var-lib-kubelet\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.143256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143221 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-etc-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-hosts-file\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143323 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5276339c-240f-48ee-afad-f81746946eb8-host-slash\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143347 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-sys-fs\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143367 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-run-ovn-kubernetes\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49b9n\" (UniqueName: \"kubernetes.io/projected/6198f643-e16b-4409-a9de-a353b91e305c-kube-api-access-49b9n\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysctl-conf\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143457 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-run\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143461 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovnkube-config\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-lib-modules\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143286 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-sys\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-os-release\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143261 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-ovn\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-systemd\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.144101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143575 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-hosts-file\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143591 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6vs\" (UniqueName: \"kubernetes.io/projected/e427f223-46d1-4317-a69e-7882713600cd-kube-api-access-nf6vs\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwkm\" (UniqueName: \"kubernetes.io/projected/5276339c-240f-48ee-afad-f81746946eb8-kube-api-access-snwkm\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-kubelet\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143661 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-systemd-units\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-env-overrides\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143696 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-lib-modules\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-log-socket\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-sys-fs\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-socket-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-registration-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143781 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5276339c-240f-48ee-afad-f81746946eb8-iptables-alerter-script\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-cni-netd\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143819 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-cni-netd\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dacf6111-b222-4313-b252-ef561ed2745d-tmp\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysconfig\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.144901 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143851 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysctl-conf\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz88l\" (UniqueName: \"kubernetes.io/projected/dacf6111-b222-4313-b252-ef561ed2745d-kube-api-access-xz88l\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7hb\" (UniqueName: \"kubernetes.io/projected/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-kube-api-access-wf7hb\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-slash\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143975 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-os-release\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-tmp-dir\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-cnibin\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144055 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-registration-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysctl-d\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.143659 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-device-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144131 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-run\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-var-lib-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144158 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysconfig\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144202 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-var-lib-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.145731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovnkube-script-lib\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144292 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-log-socket\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-systemd\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144374 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-systemd\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144092 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-systemd-units\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144426 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-systemd\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144538 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-slash\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144581 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144580 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-tmp-dir\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144593 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144233 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-socket-dir\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144064 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-kubelet\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.144665 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.144715 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs podName:6198f643-e16b-4409-a9de-a353b91e305c nodeName:}" failed. No retries permitted until 2026-07-17 18:09:52.644698626 +0000 UTC m=+3.059436646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs") pod "network-metrics-daemon-xlrp2" (UID: "6198f643-e16b-4409-a9de-a353b91e305c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:52.146557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144724 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e427f223-46d1-4317-a69e-7882713600cd-cnibin\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144769 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-sysctl-d\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144806 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144818 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-run-openvswitch\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-host\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnp9c\" (UniqueName: \"kubernetes.io/projected/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-kube-api-access-nnp9c\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144976 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-host\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-kubernetes\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.144974 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f309d6-aee5-4407-bfc2-a3f9b61cd006-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.145032 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dacf6111-b222-4313-b252-ef561ed2745d-etc-kubernetes\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.145081 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovnkube-script-lib\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.145330 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e427f223-46d1-4317-a69e-7882713600cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.145655 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dacf6111-b222-4313-b252-ef561ed2745d-etc-tuned\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.147443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.147124 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dacf6111-b222-4313-b252-ef561ed2745d-tmp\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.148111 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.147778 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03f309d6-aee5-4407-bfc2-a3f9b61cd006-ovn-node-metrics-cert\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.149160 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.149138 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:52.149160 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.149163 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:52.149365 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.149174 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4bw25 for pod openshift-network-diagnostics/network-check-target-gxfcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:52.149365 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.149241 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25 podName:6b3b6408-9e70-498c-b543-1fbb4eef10fe nodeName:}" failed. No retries permitted until 2026-07-17 18:09:52.649224457 +0000 UTC m=+3.063962484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4bw25" (UniqueName: "kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25") pod "network-check-target-gxfcg" (UID: "6b3b6408-9e70-498c-b543-1fbb4eef10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:52.151798 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.151737 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6vs\" (UniqueName: \"kubernetes.io/projected/e427f223-46d1-4317-a69e-7882713600cd-kube-api-access-nf6vs\") pod \"multus-additional-cni-plugins-24fhs\" (UID: \"e427f223-46d1-4317-a69e-7882713600cd\") " pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.152034 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.151943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lc7q\" (UniqueName: \"kubernetes.io/projected/03f309d6-aee5-4407-bfc2-a3f9b61cd006-kube-api-access-6lc7q\") pod \"ovnkube-node-txxzb\" (UID: \"03f309d6-aee5-4407-bfc2-a3f9b61cd006\") " pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.152034 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.151977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49b9n\" (UniqueName: \"kubernetes.io/projected/6198f643-e16b-4409-a9de-a353b91e305c-kube-api-access-49b9n\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:52.152567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.152545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwkm\" (UniqueName: \"kubernetes.io/projected/5276339c-240f-48ee-afad-f81746946eb8-kube-api-access-snwkm\") pod \"iptables-alerter-5hh2k\" (UID: \"5276339c-240f-48ee-afad-f81746946eb8\") " pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.153367 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.153194 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz88l\" (UniqueName: \"kubernetes.io/projected/dacf6111-b222-4313-b252-ef561ed2745d-kube-api-access-xz88l\") pod \"tuned-z6s9b\" (UID: \"dacf6111-b222-4313-b252-ef561ed2745d\") " pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.153949 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.153934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnp9c\" (UniqueName: \"kubernetes.io/projected/2d6eeff8-dd1f-4f79-b43c-dd5a564920b2-kube-api-access-nnp9c\") pod \"aws-ebs-csi-driver-node-dz8qb\" (UID: \"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.154038 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.153941 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7hb\" (UniqueName: \"kubernetes.io/projected/5b32749a-dabf-4c93-9dbd-cd822fb5ba85-kube-api-access-wf7hb\") pod \"node-resolver-52kz4\" (UID: \"5b32749a-dabf-4c93-9dbd-cd822fb5ba85\") " pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.231891 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.231857 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-btwwl" Jul 17 18:09:52.240994 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.240968 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-45tjm" Jul 17 18:09:52.248853 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.248829 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:09:52.255606 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.255578 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" Jul 17 18:09:52.263288 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.263265 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52kz4" Jul 17 18:09:52.277035 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.277003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-24fhs" Jul 17 18:09:52.285982 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.285905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5hh2k" Jul 17 18:09:52.292899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.292880 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:09:52.299850 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.299822 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" Jul 17 18:09:52.548559 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.548247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:52.548559 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.548371 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:52.548559 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.548431 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret podName:32593b2d-c97f-4e09-941c-0c2aaa602360 nodeName:}" failed. No retries permitted until 2026-07-17 18:09:53.548413987 +0000 UTC m=+3.963152013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret") pod "global-pull-secret-syncer-zv9q4" (UID: "32593b2d-c97f-4e09-941c-0c2aaa602360") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:52.648803 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.648773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:52.648985 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.648912 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:52.648985 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.648980 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs podName:6198f643-e16b-4409-a9de-a353b91e305c nodeName:}" failed. No retries permitted until 2026-07-17 18:09:53.648950086 +0000 UTC m=+4.063688091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs") pod "network-metrics-daemon-xlrp2" (UID: "6198f643-e16b-4409-a9de-a353b91e305c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:52.750149 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.750117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:52.750337 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.750250 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:52.750337 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.750273 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:52.750337 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.750286 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4bw25 for pod openshift-network-diagnostics/network-check-target-gxfcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:52.750491 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:52.750340 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25 podName:6b3b6408-9e70-498c-b543-1fbb4eef10fe nodeName:}" failed. No retries permitted until 2026-07-17 18:09:53.750327126 +0000 UTC m=+4.165065131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bw25" (UniqueName: "kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25") pod "network-check-target-gxfcg" (UID: "6b3b6408-9e70-498c-b543-1fbb4eef10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:52.967610 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.967531 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:04:51 +0000 UTC" deadline="2028-01-17 16:36:44.05567033 +0000 UTC" Jul 17 18:09:52.967610 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:52.967564 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13174h26m51.088110818s" Jul 17 18:09:53.023705 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:53.023671 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f309d6_aee5_4407_bfc2_a3f9b61cd006.slice/crio-b9c14939b574371a3c980953b4110e054a6ea02cdaa22dcf011dd619fd27a0ed WatchSource:0}: Error finding container b9c14939b574371a3c980953b4110e054a6ea02cdaa22dcf011dd619fd27a0ed: Status 404 returned error can't find the container with id b9c14939b574371a3c980953b4110e054a6ea02cdaa22dcf011dd619fd27a0ed Jul 17 18:09:53.024888 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:53.024862 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4a95c5_11f5_4b4b_a867_6a11cbd3aded.slice/crio-53fd35c78a46a5f7fc4424095e4a58b665cc3218b0f0799bf26a879ab01d9db0 WatchSource:0}: Error finding container 53fd35c78a46a5f7fc4424095e4a58b665cc3218b0f0799bf26a879ab01d9db0: Status 404 returned error can't find the container with id 53fd35c78a46a5f7fc4424095e4a58b665cc3218b0f0799bf26a879ab01d9db0 Jul 17 18:09:53.025442 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:53.025394 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode427f223_46d1_4317_a69e_7882713600cd.slice/crio-c0db065e1954b8358f32a19534d596d67c3e1216cae7a24688af02bbddf0ba1f WatchSource:0}: Error finding container c0db065e1954b8358f32a19534d596d67c3e1216cae7a24688af02bbddf0ba1f: Status 404 returned error can't find the container with id c0db065e1954b8358f32a19534d596d67c3e1216cae7a24688af02bbddf0ba1f Jul 17 18:09:53.029391 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:09:53.027446 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5276339c_240f_48ee_afad_f81746946eb8.slice/crio-f41a0ed3ccf925085d0489234e0e4711cd4db3c3f6ecd49237f9641dcd171589 WatchSource:0}: Error finding container f41a0ed3ccf925085d0489234e0e4711cd4db3c3f6ecd49237f9641dcd171589: Status 404 returned error can't find the container with id f41a0ed3ccf925085d0489234e0e4711cd4db3c3f6ecd49237f9641dcd171589 Jul 17 18:09:53.074124 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.074094 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-btwwl" event={"ID":"c0405c6f-6271-4413-aed6-8bf5750e553e","Type":"ContainerStarted","Data":"034d5df2d1b7c85937a434ff57e539ef03a195e8fcd99d252f41ad2ed7c09d0f"} Jul 17 18:09:53.075229 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.075207 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" event={"ID":"dacf6111-b222-4313-b252-ef561ed2745d","Type":"ContainerStarted","Data":"a2b591b4bb476984dc2c4570f3b088812bc7aae071844c369d9d15154994e67f"} Jul 17 18:09:53.079131 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.079104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" event={"ID":"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2","Type":"ContainerStarted","Data":"afafa0b9c2f0ec48f472d802caef22f4d5a5e8f227ca6467d297e21cc0113a80"} Jul 17 18:09:53.082293 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.082255 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerStarted","Data":"c0db065e1954b8358f32a19534d596d67c3e1216cae7a24688af02bbddf0ba1f"} Jul 17 18:09:53.083902 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.083879 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"b9c14939b574371a3c980953b4110e054a6ea02cdaa22dcf011dd619fd27a0ed"} Jul 17 18:09:53.084847 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.084824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52kz4" event={"ID":"5b32749a-dabf-4c93-9dbd-cd822fb5ba85","Type":"ContainerStarted","Data":"a50868384fe86ed1913c4f2d91370828d19055c5b93984f625d2f3635bc6ff24"} Jul 17 18:09:53.085921 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.085897 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qrr5b" event={"ID":"83445c09-b601-4b2a-b503-1d4d7138ee68","Type":"ContainerStarted","Data":"267e62b0ab70bf261241fc00003677856710211f72332c42b3954668da7ca083"} Jul 17 18:09:53.086931 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.086906 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5hh2k" event={"ID":"5276339c-240f-48ee-afad-f81746946eb8","Type":"ContainerStarted","Data":"f41a0ed3ccf925085d0489234e0e4711cd4db3c3f6ecd49237f9641dcd171589"} Jul 17 18:09:53.088018 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.088000 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-45tjm" event={"ID":"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded","Type":"ContainerStarted","Data":"53fd35c78a46a5f7fc4424095e4a58b665cc3218b0f0799bf26a879ab01d9db0"} Jul 17 18:09:53.556620 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.556319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:53.556620 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.556463 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:53.556879 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.556679 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret podName:32593b2d-c97f-4e09-941c-0c2aaa602360 nodeName:}" failed. No retries permitted until 2026-07-17 18:09:55.556657776 +0000 UTC m=+5.971395796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret") pod "global-pull-secret-syncer-zv9q4" (UID: "32593b2d-c97f-4e09-941c-0c2aaa602360") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:53.657857 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.657818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:53.658116 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.658083 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:53.658199 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.658162 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs podName:6198f643-e16b-4409-a9de-a353b91e305c nodeName:}" failed. No retries permitted until 2026-07-17 18:09:55.658141419 +0000 UTC m=+6.072879425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs") pod "network-metrics-daemon-xlrp2" (UID: "6198f643-e16b-4409-a9de-a353b91e305c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:53.758555 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.758512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:53.758737 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.758725 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:53.758801 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.758745 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:53.758801 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.758757 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4bw25 for pod openshift-network-diagnostics/network-check-target-gxfcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:53.758899 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:53.758818 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25 podName:6b3b6408-9e70-498c-b543-1fbb4eef10fe nodeName:}" failed. No retries permitted until 2026-07-17 18:09:55.75879885 +0000 UTC m=+6.173536880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bw25" (UniqueName: "kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25") pod "network-check-target-gxfcg" (UID: "6b3b6408-9e70-498c-b543-1fbb4eef10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:53.968621 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.968434 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:04:51 +0000 UTC" deadline="2028-01-15 18:07:05.272575427 +0000 UTC" Jul 17 18:09:53.968621 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:53.968493 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13127h57m11.304087248s" Jul 17 18:09:54.069341 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:54.069312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:54.069522 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:54.069445 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:09:54.069840 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:54.069823 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:54.069949 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:54.069927 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:09:54.070048 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:54.070010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:54.070109 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:54.070079 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:09:54.111294 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:54.111258 2570 generic.go:356] "Generic (PLEG): container finished" podID="51056fe6463a449526dfa7fd7c89f346" containerID="3233b929275a9b3e428d83b8e9b6834e73c943506c5eb373c206109e9f44fd68" exitCode=0 Jul 17 18:09:54.111474 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:54.111355 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" event={"ID":"51056fe6463a449526dfa7fd7c89f346","Type":"ContainerDied","Data":"3233b929275a9b3e428d83b8e9b6834e73c943506c5eb373c206109e9f44fd68"} Jul 17 18:09:54.146992 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:54.146942 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" event={"ID":"a98b89d541a53d94896a6b1c12baf188","Type":"ContainerStarted","Data":"4f07c7254f50f4b817db5479a8bf1d097ff98c2129b5b360061bcd01f5d8922a"} Jul 17 18:09:54.172370 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:54.172304 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-221.ec2.internal" podStartSLOduration=3.172282118 podStartE2EDuration="3.172282118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:09:54.171268093 +0000 UTC m=+4.586006121" watchObservedRunningTime="2026-07-17 18:09:54.172282118 +0000 UTC m=+4.587020149" Jul 17 18:09:55.171706 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:55.171656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" event={"ID":"51056fe6463a449526dfa7fd7c89f346","Type":"ContainerStarted","Data":"76bd02deb8c6db1be6eb0b4132d25ecbf50063575d91d00f11b26ae1956fe892"} Jul 17 18:09:55.190756 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:55.190693 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-221.ec2.internal" podStartSLOduration=4.190672386 podStartE2EDuration="4.190672386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:09:55.189213156 +0000 UTC m=+5.603951181" watchObservedRunningTime="2026-07-17 18:09:55.190672386 +0000 UTC m=+5.605410414" Jul 17 18:09:55.573706 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:55.573609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:55.573882 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.573791 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:55.573882 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.573856 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret podName:32593b2d-c97f-4e09-941c-0c2aaa602360 nodeName:}" failed. No retries permitted until 2026-07-17 18:09:59.573837274 +0000 UTC m=+9.988575291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret") pod "global-pull-secret-syncer-zv9q4" (UID: "32593b2d-c97f-4e09-941c-0c2aaa602360") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:55.677585 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:55.673979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:55.677585 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.674184 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:55.677585 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.674246 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs podName:6198f643-e16b-4409-a9de-a353b91e305c nodeName:}" failed. No retries permitted until 2026-07-17 18:09:59.674228015 +0000 UTC m=+10.088966035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs") pod "network-metrics-daemon-xlrp2" (UID: "6198f643-e16b-4409-a9de-a353b91e305c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:55.774634 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:55.774589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:55.774803 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.774751 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:55.774803 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.774769 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:55.774803 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.774781 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4bw25 for pod openshift-network-diagnostics/network-check-target-gxfcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:55.775023 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:55.774837 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25 podName:6b3b6408-9e70-498c-b543-1fbb4eef10fe nodeName:}" failed. No retries permitted until 2026-07-17 18:09:59.774821025 +0000 UTC m=+10.189559045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bw25" (UniqueName: "kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25") pod "network-check-target-gxfcg" (UID: "6b3b6408-9e70-498c-b543-1fbb4eef10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:56.069249 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:56.069166 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:56.069406 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:56.069301 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:09:56.069699 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:56.069677 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:56.069808 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:56.069787 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:09:56.069872 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:56.069851 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:56.069938 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:56.069920 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:09:58.067699 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:58.067665 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:58.068180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:58.067670 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:58.068180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:58.067670 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:58.068180 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:58.067879 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:09:58.068180 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:58.068039 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:09:58.068180 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:58.068141 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:09:59.609296 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:59.609258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:09:59.609783 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.609448 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:59.609783 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.609511 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret podName:32593b2d-c97f-4e09-941c-0c2aaa602360 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:07.609492695 +0000 UTC m=+18.024230707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret") pod "global-pull-secret-syncer-zv9q4" (UID: "32593b2d-c97f-4e09-941c-0c2aaa602360") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:59.710120 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:59.709835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:09:59.710120 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.709988 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:59.710120 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.710052 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs podName:6198f643-e16b-4409-a9de-a353b91e305c nodeName:}" failed. No retries permitted until 2026-07-17 18:10:07.710033387 +0000 UTC m=+18.124771412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs") pod "network-metrics-daemon-xlrp2" (UID: "6198f643-e16b-4409-a9de-a353b91e305c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:59.810938 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:09:59.810893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:09:59.811323 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.811167 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:59.811323 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.811198 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:59.811323 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.811212 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4bw25 for pod openshift-network-diagnostics/network-check-target-gxfcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:59.811323 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:09:59.811277 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25 podName:6b3b6408-9e70-498c-b543-1fbb4eef10fe nodeName:}" failed. No retries permitted until 2026-07-17 18:10:07.811256256 +0000 UTC m=+18.225994286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bw25" (UniqueName: "kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25") pod "network-check-target-gxfcg" (UID: "6b3b6408-9e70-498c-b543-1fbb4eef10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:00.069206 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:00.068375 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:00.069206 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:00.068495 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:00.069206 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:00.068877 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:00.069206 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:00.068997 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:00.069206 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:00.069058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:00.069206 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:00.069124 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:02.066537 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:02.066504 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:02.067027 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:02.066503 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:02.067027 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:02.066656 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:02.067027 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:02.066743 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:02.067027 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:02.066504 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:02.067027 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:02.066844 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:04.066256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:04.066221 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:04.066699 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:04.066362 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:04.066699 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:04.066221 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:04.066699 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:04.066218 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:04.066699 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:04.066470 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:04.066699 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:04.066556 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:06.067040 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:06.067006 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:06.067040 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:06.067039 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:06.067622 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:06.067007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:06.067622 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:06.067128 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:06.067622 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:06.067220 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:06.067622 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:06.067286 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:07.674726 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:07.674629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:07.675234 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.674811 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:07.675234 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.674898 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret podName:32593b2d-c97f-4e09-941c-0c2aaa602360 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:23.674874567 +0000 UTC m=+34.089612589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret") pod "global-pull-secret-syncer-zv9q4" (UID: "32593b2d-c97f-4e09-941c-0c2aaa602360") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:07.775859 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:07.775821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:07.776045 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.775943 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:07.776045 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.776029 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs podName:6198f643-e16b-4409-a9de-a353b91e305c nodeName:}" failed. No retries permitted until 2026-07-17 18:10:23.776013379 +0000 UTC m=+34.190751405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs") pod "network-metrics-daemon-xlrp2" (UID: "6198f643-e16b-4409-a9de-a353b91e305c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:07.877089 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:07.877057 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:07.877267 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.877195 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:10:07.877267 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.877212 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:10:07.877267 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.877223 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4bw25 for pod openshift-network-diagnostics/network-check-target-gxfcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:07.877393 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:07.877276 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25 podName:6b3b6408-9e70-498c-b543-1fbb4eef10fe nodeName:}" failed. No retries permitted until 2026-07-17 18:10:23.877263141 +0000 UTC m=+34.292001146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bw25" (UniqueName: "kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25") pod "network-check-target-gxfcg" (UID: "6b3b6408-9e70-498c-b543-1fbb4eef10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:08.067255 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:08.067173 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:08.067255 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:08.067202 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:08.067466 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:08.067173 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:08.067466 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:08.067301 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:08.067466 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:08.067407 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:08.067636 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:08.067508 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:10.067795 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:10.067761 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:10.068270 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:10.067868 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:10.068270 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:10.067949 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:10.068270 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:10.068070 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:10.068270 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:10.068120 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:10.068270 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:10.068240 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:12.066773 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.066526 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:12.067164 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:12.066882 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:12.067164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.066669 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:12.067164 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:12.066996 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:12.067164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.066554 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:12.067164 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:12.067079 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:12.211565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.211539 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52kz4" event={"ID":"5b32749a-dabf-4c93-9dbd-cd822fb5ba85","Type":"ContainerStarted","Data":"604903dc286f3944a6b2745ee1f9b10c25b2bf9b8e5f137bd78da3acf5555221"} Jul 17 18:10:12.213534 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.213510 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qrr5b" event={"ID":"83445c09-b601-4b2a-b503-1d4d7138ee68","Type":"ContainerStarted","Data":"48cd60ffd3fcbd1376a942e70c7bf77a6bc73283f5c26469781890ef98be1e15"} Jul 17 18:10:12.215065 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.215034 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-btwwl" event={"ID":"c0405c6f-6271-4413-aed6-8bf5750e553e","Type":"ContainerStarted","Data":"5d18958706ba0a6826dcb0a39537239c4a88aa13d324161dd5a17febd368d580"} Jul 17 18:10:12.216538 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.216513 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" event={"ID":"dacf6111-b222-4313-b252-ef561ed2745d","Type":"ContainerStarted","Data":"cd9592ed025f12f34ddd8353791f2acf236bc4c1f9a07ec5530e7b22cf535d5e"} Jul 17 18:10:12.218463 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.218435 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" event={"ID":"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2","Type":"ContainerStarted","Data":"b4e45fe85bc4bf8eb38bd4786ed5bb93a751935192708176597062bd6c2bc40b"} Jul 17 18:10:12.219938 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.219911 2570 generic.go:356] "Generic (PLEG): container finished" podID="e427f223-46d1-4317-a69e-7882713600cd" containerID="0191e57e9d33d2b99de3f0104165a61e06fbca41e61fa02fe5812c9d25a4f62a" exitCode=0 Jul 17 18:10:12.220046 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.219984 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerDied","Data":"0191e57e9d33d2b99de3f0104165a61e06fbca41e61fa02fe5812c9d25a4f62a"} Jul 17 18:10:12.222014 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.221998 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:10:12.222368 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.222313 2570 generic.go:356] "Generic (PLEG): container finished" podID="03f309d6-aee5-4407-bfc2-a3f9b61cd006" containerID="54d160c7eac23fc754d6dd79c17a9d1fbda3349a3f0df7ba18aafd4d08d9d481" exitCode=1 Jul 17 18:10:12.222368 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.222337 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerDied","Data":"54d160c7eac23fc754d6dd79c17a9d1fbda3349a3f0df7ba18aafd4d08d9d481"} Jul 17 18:10:12.222480 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.222395 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"5d36f2631c27bcc8996e72f54a1df917c353f4b121870f3c7eebbc232a0c513e"} Jul 17 18:10:12.229658 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.229544 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-52kz4" podStartSLOduration=3.386078135 podStartE2EDuration="22.22953165s" totalImagesPullingTime="18.843453515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:12.228844078 +0000 UTC m=+22.643582106" watchObservedRunningTime="2026-07-17 18:10:12.22953165 +0000 UTC m=+22.644269677" Jul 17 18:10:12.242818 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.242771 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-btwwl" podStartSLOduration=3.400827896 podStartE2EDuration="22.242757572s" totalImagesPullingTime="18.841929676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:12.241975062 +0000 UTC m=+22.656713090" watchObservedRunningTime="2026-07-17 18:10:12.242757572 +0000 UTC m=+22.657495599" Jul 17 18:10:12.280922 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.280860 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z6s9b" podStartSLOduration=3.43698051 podStartE2EDuration="22.280839676s" totalImagesPullingTime="18.843859166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:12.279942026 +0000 UTC m=+22.694680053" watchObservedRunningTime="2026-07-17 18:10:12.280839676 +0000 UTC m=+22.695577704" Jul 17 18:10:12.295297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:12.295260 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qrr5b" podStartSLOduration=7.965125901 podStartE2EDuration="22.295245856s" totalImagesPullingTime="14.330119955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:12.294782123 +0000 UTC m=+22.709520151" watchObservedRunningTime="2026-07-17 18:10:12.295245856 +0000 UTC m=+22.709983882" Jul 17 18:10:13.229061 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:13.228914 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:10:13.229599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:13.229469 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"3494c8641dfe0fffe97603ee5d3a2319e69bea58c3c0f44784677ecac9430be3"} Jul 17 18:10:13.229599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:13.229592 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"16736659cf8baf01cb79e8977fca347a62b6de89214ae79fa1370f6386dbcf5c"} Jul 17 18:10:13.229697 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:13.229647 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"47555e6f2d038d82507a23f5883e00df53a37aa79eafb6c3658088b2ab33eb3a"} Jul 17 18:10:13.229745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:13.229694 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"52d82e1605739a9a84ce86313a574088db50ebb82f819fc95987933e09d79897"} Jul 17 18:10:13.231751 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:13.231725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-45tjm" event={"ID":"2d4a95c5-11f5-4b4b-a867-6a11cbd3aded","Type":"ContainerStarted","Data":"3612e01c269522390f1061d77be07f0ac5db2c13d8eb242d3883a8b244a858e3"} Jul 17 18:10:13.521542 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:13.521515 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 18:10:14.000328 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.000029 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T18:10:13.521536571Z","UUID":"61e8bba7-57b7-4d3c-922f-2634f29cf807","Handler":null,"Name":"","Endpoint":""} Jul 17 18:10:14.001791 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.001769 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 18:10:14.001930 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.001814 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 18:10:14.067686 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.067602 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:14.067686 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.067628 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:14.067922 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:14.067730 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:14.068068 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.068050 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:14.068180 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:14.068157 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:14.068289 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:14.068244 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:14.235558 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.235517 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5hh2k" event={"ID":"5276339c-240f-48ee-afad-f81746946eb8","Type":"ContainerStarted","Data":"ba2941e412ea837b17de7de23e180893fcbb6ba50b705ad10eb7a1a579d0852f"} Jul 17 18:10:14.237566 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.237521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" event={"ID":"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2","Type":"ContainerStarted","Data":"75626f998ca2e4c35fb21e7033fdd28808ee55ac96f1aaf6a08c74e3754ddb05"} Jul 17 18:10:14.252208 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.252118 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-45tjm" podStartSLOduration=5.079457583 podStartE2EDuration="24.252098929s" totalImagesPullingTime="19.172641346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:13.254912609 +0000 UTC m=+23.669650636" watchObservedRunningTime="2026-07-17 18:10:14.252098929 +0000 UTC m=+24.666836958" Jul 17 18:10:14.252657 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.252626 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5hh2k" podStartSLOduration=5.432402176 podStartE2EDuration="24.252619457s" totalImagesPullingTime="18.820217281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:14.25203483 +0000 UTC m=+24.666772859" watchObservedRunningTime="2026-07-17 18:10:14.252619457 +0000 UTC m=+24.667357486" Jul 17 18:10:14.551002 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.550902 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:10:14.551600 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:14.551582 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:10:15.242153 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:15.242064 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" event={"ID":"2d6eeff8-dd1f-4f79-b43c-dd5a564920b2","Type":"ContainerStarted","Data":"2ce8c45f41eb0635272d4e7ba03e133e689575eb84d94340662c90b05fef61e2"} Jul 17 18:10:15.245708 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:15.245683 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:10:15.246169 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:15.246143 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"eff2cea627662f70d8d40d2f986fe7640d000c9e4bf1afd03e8888689ac529f5"} Jul 17 18:10:15.246547 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:15.246526 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:10:15.247073 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:15.247049 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qrr5b" Jul 17 18:10:15.260466 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:15.260421 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz8qb" podStartSLOduration=3.90697563 podStartE2EDuration="25.26040571s" totalImagesPullingTime="21.35343008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:15.260054066 +0000 UTC m=+25.674792094" watchObservedRunningTime="2026-07-17 18:10:15.26040571 +0000 UTC m=+25.675143736" Jul 17 18:10:16.066413 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:16.066166 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:16.066582 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:16.066518 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:16.066582 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:16.066209 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:16.066683 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:16.066624 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:16.066683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:16.066166 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:16.066752 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:16.066713 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:17.253809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:17.253593 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:10:17.254608 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:17.254129 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"a4be9ce4f2c9e03d25a2b59df6dd90355650b7a5ada995ef4eab7aa72e38e59b"} Jul 17 18:10:17.254608 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:17.254358 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:10:17.254608 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:17.254535 2570 scope.go:117] "RemoveContainer" containerID="54d160c7eac23fc754d6dd79c17a9d1fbda3349a3f0df7ba18aafd4d08d9d481" Jul 17 18:10:17.256085 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:17.256063 2570 generic.go:356] "Generic (PLEG): container finished" podID="e427f223-46d1-4317-a69e-7882713600cd" containerID="22a8cd9f60c89eece12c6aacea5b0973a274c741d13674ad460d483ca9ccdc91" exitCode=0 Jul 17 18:10:17.256202 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:17.256156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerDied","Data":"22a8cd9f60c89eece12c6aacea5b0973a274c741d13674ad460d483ca9ccdc91"} Jul 17 18:10:17.270751 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:17.270724 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:10:18.066161 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.066123 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:18.066337 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.066227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:18.066337 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.066256 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:18.066337 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:18.066296 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:18.066337 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:18.066292 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:18.066533 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:18.066357 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:18.263886 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.263812 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:10:18.264401 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.264374 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" event={"ID":"03f309d6-aee5-4407-bfc2-a3f9b61cd006","Type":"ContainerStarted","Data":"996af8954bae2e1118ccbddfd25846acd809f9f9bbce95f695545d0e131603ee"} Jul 17 18:10:18.265534 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.265402 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:10:18.265534 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.265488 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:10:18.287296 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.287261 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:10:18.324197 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:18.324139 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" podStartSLOduration=9.416948476 podStartE2EDuration="28.32411815s" totalImagesPullingTime="18.907169674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:18.298366765 +0000 UTC m=+28.713104797" watchObservedRunningTime="2026-07-17 18:10:18.32411815 +0000 UTC m=+28.738856181" Jul 17 18:10:19.239042 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.238972 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zv9q4"] Jul 17 18:10:19.239201 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.239125 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:19.239264 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:19.239239 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:19.252353 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.252320 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xlrp2"] Jul 17 18:10:19.252508 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.252455 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:19.252587 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:19.252564 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:19.253298 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.253273 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gxfcg"] Jul 17 18:10:19.253399 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.253354 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:19.253434 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:19.253413 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:19.269454 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.269419 2570 generic.go:356] "Generic (PLEG): container finished" podID="e427f223-46d1-4317-a69e-7882713600cd" containerID="8847067f0df2a97d56ac77a964930d8da76c2f63862e2b8d125eaabd66304bc5" exitCode=0 Jul 17 18:10:19.269860 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:19.269524 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerDied","Data":"8847067f0df2a97d56ac77a964930d8da76c2f63862e2b8d125eaabd66304bc5"} Jul 17 18:10:21.067089 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:21.066886 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:21.067453 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:21.066899 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:21.067453 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:21.067160 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:21.067453 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:21.067247 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:21.067453 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:21.066938 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:21.067453 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:21.067337 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:21.276576 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:21.276493 2570 generic.go:356] "Generic (PLEG): container finished" podID="e427f223-46d1-4317-a69e-7882713600cd" containerID="aac4b701d496067cb9b63bc02034f9d31866333fb0930d4c447d364f7c5cab2b" exitCode=0 Jul 17 18:10:21.276576 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:21.276551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerDied","Data":"aac4b701d496067cb9b63bc02034f9d31866333fb0930d4c447d364f7c5cab2b"} Jul 17 18:10:23.066248 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:23.066213 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:23.066887 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:23.066258 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:23.066887 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:23.066321 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:23.066887 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.066340 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxfcg" podUID="6b3b6408-9e70-498c-b543-1fbb4eef10fe" Jul 17 18:10:23.066887 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.066417 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlrp2" podUID="6198f643-e16b-4409-a9de-a353b91e305c" Jul 17 18:10:23.066887 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.066501 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9q4" podUID="32593b2d-c97f-4e09-941c-0c2aaa602360" Jul 17 18:10:23.694180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:23.694147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:23.694383 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.694282 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:23.694383 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.694343 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret podName:32593b2d-c97f-4e09-941c-0c2aaa602360 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:55.694325582 +0000 UTC m=+66.109063600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret") pod "global-pull-secret-syncer-zv9q4" (UID: "32593b2d-c97f-4e09-941c-0c2aaa602360") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:23.794947 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:23.794904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:23.795142 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.795084 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:23.795214 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.795163 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs podName:6198f643-e16b-4409-a9de-a353b91e305c nodeName:}" failed. No retries permitted until 2026-07-17 18:10:55.795142894 +0000 UTC m=+66.209880912 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs") pod "network-metrics-daemon-xlrp2" (UID: "6198f643-e16b-4409-a9de-a353b91e305c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:23.896140 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:23.896093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:23.896327 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.896292 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:10:23.896327 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.896319 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:10:23.896437 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.896332 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4bw25 for pod openshift-network-diagnostics/network-check-target-gxfcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:23.896774 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:23.896733 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25 podName:6b3b6408-9e70-498c-b543-1fbb4eef10fe nodeName:}" failed. No retries permitted until 2026-07-17 18:10:55.896381181 +0000 UTC m=+66.311119204 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bw25" (UniqueName: "kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25") pod "network-check-target-gxfcg" (UID: "6b3b6408-9e70-498c-b543-1fbb4eef10fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:24.812515 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.812479 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-221.ec2.internal" event="NodeReady" Jul 17 18:10:24.813063 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.812658 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 18:10:24.848055 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.847971 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86fc465cf-mhntk"] Jul 17 18:10:24.852444 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.852416 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:24.855876 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.855854 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 18:10:24.856013 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.855895 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 18:10:24.856514 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.856491 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ksssl\"" Jul 17 18:10:24.856763 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.856709 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 18:10:24.861610 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.861436 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f8dn9"] Jul 17 18:10:24.862772 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.862751 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 18:10:24.864558 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.864537 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rhq8s"] Jul 17 18:10:24.864773 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.864754 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:24.867133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.867113 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 18:10:24.867235 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.867162 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vkv2q\"" Jul 17 18:10:24.867301 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.867116 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 18:10:24.868122 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.868101 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86fc465cf-mhntk"] Jul 17 18:10:24.868249 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.868212 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:24.870604 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.870582 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 18:10:24.871080 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.870859 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 18:10:24.871080 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.870913 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 18:10:24.871080 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.870868 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7247w\"" Jul 17 18:10:24.873547 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.873527 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f8dn9"] Jul 17 18:10:24.878473 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:24.878455 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rhq8s"] Jul 17 18:10:25.003884 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.003855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.004082 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.003888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-image-registry-private-configuration\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004082 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.003912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004082 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.003932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-certificates\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004082 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004013 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae35be8c-027a-4136-a56f-93326f08a0ee-ca-trust-extracted\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004082 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004058 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-installation-pull-secrets\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-kube-api-access-bmgls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d905cff9-94ae-47e1-9c19-490ebe8a40e3-tmp-dir\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.004348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004188 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d905cff9-94ae-47e1-9c19-490ebe8a40e3-config-volume\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.004348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004212 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-bound-sa-token\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004239 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhw62\" (UniqueName: \"kubernetes.io/projected/3ac4f76b-38c8-4a90-bb4d-da0959212e44-kube-api-access-lhw62\") pod \"ingress-canary-rhq8s\" (UID: \"3ac4f76b-38c8-4a90-bb4d-da0959212e44\") " pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:25.004348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-trusted-ca\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.004568 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004358 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac4f76b-38c8-4a90-bb4d-da0959212e44-cert\") pod \"ingress-canary-rhq8s\" (UID: \"3ac4f76b-38c8-4a90-bb4d-da0959212e44\") " pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:25.004568 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.004402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkds\" (UniqueName: \"kubernetes.io/projected/d905cff9-94ae-47e1-9c19-490ebe8a40e3-kube-api-access-pfkds\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.066236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.066208 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:25.066236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.066228 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:25.066468 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.066227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:25.070099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.069997 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nj845\"" Jul 17 18:10:25.070099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.069898 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:10:25.070099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.070028 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:10:25.070099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.070036 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:10:25.070412 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.070172 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:10:25.070412 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.070363 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47jdr\"" Jul 17 18:10:25.105669 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.105603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d905cff9-94ae-47e1-9c19-490ebe8a40e3-tmp-dir\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.105669 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.105639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d905cff9-94ae-47e1-9c19-490ebe8a40e3-config-volume\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.105669 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.105658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-bound-sa-token\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.105915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.105792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhw62\" (UniqueName: \"kubernetes.io/projected/3ac4f76b-38c8-4a90-bb4d-da0959212e44-kube-api-access-lhw62\") pod \"ingress-canary-rhq8s\" (UID: \"3ac4f76b-38c8-4a90-bb4d-da0959212e44\") " pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:25.105915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.105883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-trusted-ca\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.105915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.105910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac4f76b-38c8-4a90-bb4d-da0959212e44-cert\") pod \"ingress-canary-rhq8s\" (UID: \"3ac4f76b-38c8-4a90-bb4d-da0959212e44\") " pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:25.106072 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.105992 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d905cff9-94ae-47e1-9c19-490ebe8a40e3-tmp-dir\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkds\" (UniqueName: \"kubernetes.io/projected/d905cff9-94ae-47e1-9c19-490ebe8a40e3-kube-api-access-pfkds\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-image-registry-private-configuration\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106293 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-certificates\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106305 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d905cff9-94ae-47e1-9c19-490ebe8a40e3-config-volume\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106323 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae35be8c-027a-4136-a56f-93326f08a0ee-ca-trust-extracted\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.106310 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-installation-pull-secrets\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.106364 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.106377 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86fc465cf-mhntk: secret "image-registry-tls" not found Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.106386 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls podName:d905cff9-94ae-47e1-9c19-490ebe8a40e3 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:25.606370233 +0000 UTC m=+36.021108237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls") pod "dns-default-f8dn9" (UID: "d905cff9-94ae-47e1-9c19-490ebe8a40e3") : secret "dns-default-metrics-tls" not found Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-kube-api-access-bmgls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.106423 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls podName:ae35be8c-027a-4136-a56f-93326f08a0ee nodeName:}" failed. No retries permitted until 2026-07-17 18:10:25.606407179 +0000 UTC m=+36.021145200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls") pod "image-registry-86fc465cf-mhntk" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee") : secret "image-registry-tls" not found Jul 17 18:10:25.106779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106741 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae35be8c-027a-4136-a56f-93326f08a0ee-ca-trust-extracted\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.107518 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106899 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-certificates\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.107518 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.106899 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-trusted-ca\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.110518 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.110498 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-installation-pull-secrets\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.110629 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.110497 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-image-registry-private-configuration\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.110676 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.110631 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac4f76b-38c8-4a90-bb4d-da0959212e44-cert\") pod \"ingress-canary-rhq8s\" (UID: \"3ac4f76b-38c8-4a90-bb4d-da0959212e44\") " pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:25.114746 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.114705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhw62\" (UniqueName: \"kubernetes.io/projected/3ac4f76b-38c8-4a90-bb4d-da0959212e44-kube-api-access-lhw62\") pod \"ingress-canary-rhq8s\" (UID: \"3ac4f76b-38c8-4a90-bb4d-da0959212e44\") " pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:25.114861 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.114839 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkds\" (UniqueName: \"kubernetes.io/projected/d905cff9-94ae-47e1-9c19-490ebe8a40e3-kube-api-access-pfkds\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.115109 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.115086 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-kube-api-access-bmgls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.115300 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.115282 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-bound-sa-token\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.183420 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.183388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rhq8s" Jul 17 18:10:25.572044 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.571841 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rhq8s"] Jul 17 18:10:25.575540 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:25.575510 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac4f76b_38c8_4a90_bb4d_da0959212e44.slice/crio-c5c2b8a8d81af63eea957d53e8e3a5b36bbbb8adf4012c204648dcb0886921e3 WatchSource:0}: Error finding container c5c2b8a8d81af63eea957d53e8e3a5b36bbbb8adf4012c204648dcb0886921e3: Status 404 returned error can't find the container with id c5c2b8a8d81af63eea957d53e8e3a5b36bbbb8adf4012c204648dcb0886921e3 Jul 17 18:10:25.610323 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.610300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:25.610402 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:25.610332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:25.610448 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.610437 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:10:25.610448 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.610447 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86fc465cf-mhntk: secret "image-registry-tls" not found Jul 17 18:10:25.610511 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.610447 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:25.610511 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.610496 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls podName:ae35be8c-027a-4136-a56f-93326f08a0ee nodeName:}" failed. No retries permitted until 2026-07-17 18:10:26.610482156 +0000 UTC m=+37.025220161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls") pod "image-registry-86fc465cf-mhntk" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee") : secret "image-registry-tls" not found Jul 17 18:10:25.610511 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:25.610508 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls podName:d905cff9-94ae-47e1-9c19-490ebe8a40e3 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:26.610501964 +0000 UTC m=+37.025239968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls") pod "dns-default-f8dn9" (UID: "d905cff9-94ae-47e1-9c19-490ebe8a40e3") : secret "dns-default-metrics-tls" not found Jul 17 18:10:26.293030 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:26.292993 2570 generic.go:356] "Generic (PLEG): container finished" podID="e427f223-46d1-4317-a69e-7882713600cd" containerID="2c5fd0731bab536faa469807215924ef44c92396a886ecc9e676c4596f7e77f1" exitCode=0 Jul 17 18:10:26.293606 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:26.293074 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerDied","Data":"2c5fd0731bab536faa469807215924ef44c92396a886ecc9e676c4596f7e77f1"} Jul 17 18:10:26.294298 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:26.294271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rhq8s" event={"ID":"3ac4f76b-38c8-4a90-bb4d-da0959212e44","Type":"ContainerStarted","Data":"c5c2b8a8d81af63eea957d53e8e3a5b36bbbb8adf4012c204648dcb0886921e3"} Jul 17 18:10:26.618280 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:26.618241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:26.618447 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:26.618288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:26.618447 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:26.618409 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:10:26.618447 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:26.618429 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86fc465cf-mhntk: secret "image-registry-tls" not found Jul 17 18:10:26.618655 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:26.618486 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls podName:ae35be8c-027a-4136-a56f-93326f08a0ee nodeName:}" failed. No retries permitted until 2026-07-17 18:10:28.618468033 +0000 UTC m=+39.033206041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls") pod "image-registry-86fc465cf-mhntk" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee") : secret "image-registry-tls" not found Jul 17 18:10:26.618655 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:26.618408 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:26.618655 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:26.618547 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls podName:d905cff9-94ae-47e1-9c19-490ebe8a40e3 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:28.618532254 +0000 UTC m=+39.033270275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls") pod "dns-default-f8dn9" (UID: "d905cff9-94ae-47e1-9c19-490ebe8a40e3") : secret "dns-default-metrics-tls" not found Jul 17 18:10:27.301001 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:27.300946 2570 generic.go:356] "Generic (PLEG): container finished" podID="e427f223-46d1-4317-a69e-7882713600cd" containerID="b047deda47dd1dae0bf6456f1f2e5de06b409a72461f6755e9335bb71ebfe961" exitCode=0 Jul 17 18:10:27.301396 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:27.301035 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerDied","Data":"b047deda47dd1dae0bf6456f1f2e5de06b409a72461f6755e9335bb71ebfe961"} Jul 17 18:10:28.305675 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:28.305439 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rhq8s" event={"ID":"3ac4f76b-38c8-4a90-bb4d-da0959212e44","Type":"ContainerStarted","Data":"3100ec5166902dfb6baac270e2d0216be87bad1fbc0cd31379888e568a9f7bf4"} Jul 17 18:10:28.309249 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:28.309221 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24fhs" event={"ID":"e427f223-46d1-4317-a69e-7882713600cd","Type":"ContainerStarted","Data":"aec7709862737719dff8cce432c8b68d0fd1ddf14319ee90afeac464b1769fc7"} Jul 17 18:10:28.320682 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:28.320639 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rhq8s" podStartSLOduration=2.200416684 podStartE2EDuration="4.32062542s" totalImagesPullingTime="2.120208736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:28.319855076 +0000 UTC m=+38.734593102" watchObservedRunningTime="2026-07-17 18:10:28.32062542 +0000 UTC m=+38.735363445" Jul 17 18:10:28.340596 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:28.340545 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-24fhs" podStartSLOduration=8.627246534 podStartE2EDuration="38.340523306s" totalImagesPullingTime="29.713276772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:28.33957763 +0000 UTC m=+38.754315658" watchObservedRunningTime="2026-07-17 18:10:28.340523306 +0000 UTC m=+38.755261365" Jul 17 18:10:28.634590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:28.634491 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:28.634590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:28.634533 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:28.634812 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:28.634641 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:28.634812 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:28.634700 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls podName:d905cff9-94ae-47e1-9c19-490ebe8a40e3 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:32.634686094 +0000 UTC m=+43.049424099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls") pod "dns-default-f8dn9" (UID: "d905cff9-94ae-47e1-9c19-490ebe8a40e3") : secret "dns-default-metrics-tls" not found Jul 17 18:10:28.634812 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:28.634643 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:10:28.634812 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:28.634720 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86fc465cf-mhntk: secret "image-registry-tls" not found Jul 17 18:10:28.634812 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:28.634752 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls podName:ae35be8c-027a-4136-a56f-93326f08a0ee nodeName:}" failed. No retries permitted until 2026-07-17 18:10:32.634744314 +0000 UTC m=+43.049482320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls") pod "image-registry-86fc465cf-mhntk" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee") : secret "image-registry-tls" not found Jul 17 18:10:32.660369 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.660326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:32.660369 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.660372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:32.661129 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.660484 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:10:32.661129 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.660489 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:32.661129 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.660563 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls podName:d905cff9-94ae-47e1-9c19-490ebe8a40e3 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:40.66054237 +0000 UTC m=+51.075280394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls") pod "dns-default-f8dn9" (UID: "d905cff9-94ae-47e1-9c19-490ebe8a40e3") : secret "dns-default-metrics-tls" not found Jul 17 18:10:32.661129 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.660495 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86fc465cf-mhntk: secret "image-registry-tls" not found Jul 17 18:10:32.661129 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.660622 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls podName:ae35be8c-027a-4136-a56f-93326f08a0ee nodeName:}" failed. No retries permitted until 2026-07-17 18:10:40.660608752 +0000 UTC m=+51.075346757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls") pod "image-registry-86fc465cf-mhntk" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee") : secret "image-registry-tls" not found Jul 17 18:10:32.672531 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.672497 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z9ngn"] Jul 17 18:10:32.702646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.702614 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z9ngn"] Jul 17 18:10:32.702803 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.702752 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.705608 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.705550 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 18:10:32.705735 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.705711 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 18:10:32.705881 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.705802 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 18:10:32.706058 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.706040 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 18:10:32.706599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.706586 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-85j6j\"" Jul 17 18:10:32.862407 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.862373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58f6e97-8740-4378-8965-a3d69b83e938-crio-socket\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.862407 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.862414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.862639 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.862431 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp94d\" (UniqueName: \"kubernetes.io/projected/e58f6e97-8740-4378-8965-a3d69b83e938-kube-api-access-zp94d\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.862639 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.862469 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.862639 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.862579 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6e97-8740-4378-8965-a3d69b83e938-data-volume\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.963865 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.963833 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6e97-8740-4378-8965-a3d69b83e938-data-volume\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.964031 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.963875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58f6e97-8740-4378-8965-a3d69b83e938-crio-socket\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.964031 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.963895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.964031 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.963913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp94d\" (UniqueName: \"kubernetes.io/projected/e58f6e97-8740-4378-8965-a3d69b83e938-kube-api-access-zp94d\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.964031 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.964008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.964233 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.964083 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:10:32.964233 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.964093 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58f6e97-8740-4378-8965-a3d69b83e938-crio-socket\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.964233 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.964145 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:32.964233 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.964155 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:33.464134902 +0000 UTC m=+43.878872911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : configmap "kube-rbac-proxy" not found Jul 17 18:10:32.964233 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.964171 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f6e97-8740-4378-8965-a3d69b83e938-data-volume\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:32.964233 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:32.964219 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:33.464200854 +0000 UTC m=+43.878938858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:32.981377 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:32.981346 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp94d\" (UniqueName: \"kubernetes.io/projected/e58f6e97-8740-4378-8965-a3d69b83e938-kube-api-access-zp94d\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:33.467698 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:33.467661 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:33.467848 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:33.467720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:33.467848 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:33.467792 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:10:33.467848 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:33.467834 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:33.467981 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:33.467863 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:34.46784369 +0000 UTC m=+44.882581709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : configmap "kube-rbac-proxy" not found Jul 17 18:10:33.467981 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:33.467879 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:34.467871411 +0000 UTC m=+44.882609417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:34.476470 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:34.476427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:34.476903 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:34.476498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:34.476903 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:34.476553 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:10:34.476903 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:34.476627 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:36.476606547 +0000 UTC m=+46.891344555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : configmap "kube-rbac-proxy" not found Jul 17 18:10:34.476903 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:34.476662 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:34.476903 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:34.476710 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:36.476696942 +0000 UTC m=+46.891434963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:35.182121 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.182093 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52kz4_5b32749a-dabf-4c93-9dbd-cd822fb5ba85/dns-node-resolver/0.log" Jul 17 18:10:35.413892 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.413859 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-6mjt5"] Jul 17 18:10:35.448921 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.448837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-6mjt5"] Jul 17 18:10:35.449084 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.448980 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.451843 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.451819 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 17 18:10:35.451993 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.451928 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lqfpd\"" Jul 17 18:10:35.453025 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.453005 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 17 18:10:35.453025 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.453025 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 17 18:10:35.453148 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.453040 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 17 18:10:35.583317 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.583283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7e01ac18-1072-4cd6-aaab-46ce574858d3-signing-cabundle\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.583689 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.583329 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7e01ac18-1072-4cd6-aaab-46ce574858d3-signing-key\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.583689 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.583390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfm79\" (UniqueName: \"kubernetes.io/projected/7e01ac18-1072-4cd6-aaab-46ce574858d3-kube-api-access-sfm79\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.684036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.684003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7e01ac18-1072-4cd6-aaab-46ce574858d3-signing-cabundle\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.684198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.684057 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7e01ac18-1072-4cd6-aaab-46ce574858d3-signing-key\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.684198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.684090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfm79\" (UniqueName: \"kubernetes.io/projected/7e01ac18-1072-4cd6-aaab-46ce574858d3-kube-api-access-sfm79\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.684659 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.684639 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7e01ac18-1072-4cd6-aaab-46ce574858d3-signing-cabundle\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.686445 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.686423 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7e01ac18-1072-4cd6-aaab-46ce574858d3-signing-key\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.691617 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.691592 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfm79\" (UniqueName: \"kubernetes.io/projected/7e01ac18-1072-4cd6-aaab-46ce574858d3-kube-api-access-sfm79\") pod \"service-ca-58598695f7-6mjt5\" (UID: \"7e01ac18-1072-4cd6-aaab-46ce574858d3\") " pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.757727 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.757653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-6mjt5" Jul 17 18:10:35.887919 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:35.887886 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-6mjt5"] Jul 17 18:10:35.891982 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:35.891943 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e01ac18_1072_4cd6_aaab_46ce574858d3.slice/crio-f8b1f2316f06d17034604aa3b3a0398da5c8750f8e138d3fd7276e84007a0663 WatchSource:0}: Error finding container f8b1f2316f06d17034604aa3b3a0398da5c8750f8e138d3fd7276e84007a0663: Status 404 returned error can't find the container with id f8b1f2316f06d17034604aa3b3a0398da5c8750f8e138d3fd7276e84007a0663 Jul 17 18:10:36.183646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:36.183618 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-btwwl_c0405c6f-6271-4413-aed6-8bf5750e553e/node-ca/0.log" Jul 17 18:10:36.329839 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:36.329806 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-6mjt5" event={"ID":"7e01ac18-1072-4cd6-aaab-46ce574858d3","Type":"ContainerStarted","Data":"f8b1f2316f06d17034604aa3b3a0398da5c8750f8e138d3fd7276e84007a0663"} Jul 17 18:10:36.489775 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:36.489692 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:36.489949 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:36.489802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:36.489949 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:36.489876 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:36.490102 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:36.489976 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:40.489934533 +0000 UTC m=+50.904672552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:36.490356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:36.490331 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f6e97-8740-4378-8965-a3d69b83e938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:37.185249 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:37.185219 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rhq8s_3ac4f76b-38c8-4a90-bb4d-da0959212e44/serve-healthcheck-canary/0.log" Jul 17 18:10:39.339156 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:39.339121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-6mjt5" event={"ID":"7e01ac18-1072-4cd6-aaab-46ce574858d3","Type":"ContainerStarted","Data":"bf8d32f9531d81a531b51036b6f77460f9d309bdbb13fef619c9eb8c538c7b63"} Jul 17 18:10:39.355265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:39.355217 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-6mjt5" podStartSLOduration=1.7982908819999999 podStartE2EDuration="4.355200946s" totalImagesPullingTime="2.556910064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:39.35404933 +0000 UTC m=+49.768787356" watchObservedRunningTime="2026-07-17 18:10:39.355200946 +0000 UTC m=+49.769938972" Jul 17 18:10:40.519621 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:40.519582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:40.520218 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:40.519725 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:40.520462 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:40.520437 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls podName:e58f6e97-8740-4378-8965-a3d69b83e938 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:48.520411544 +0000 UTC m=+58.935149557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z9ngn" (UID: "e58f6e97-8740-4378-8965-a3d69b83e938") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:40.721629 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:40.721577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:40.721629 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:40.721629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:40.722058 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:40.722024 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:40.722231 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:40.722220 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls podName:d905cff9-94ae-47e1-9c19-490ebe8a40e3 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:56.722197966 +0000 UTC m=+67.136935986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls") pod "dns-default-f8dn9" (UID: "d905cff9-94ae-47e1-9c19-490ebe8a40e3") : secret "dns-default-metrics-tls" not found Jul 17 18:10:40.722334 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:40.722030 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:10:40.722422 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:40.722414 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86fc465cf-mhntk: secret "image-registry-tls" not found Jul 17 18:10:40.722530 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:10:40.722523 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls podName:ae35be8c-027a-4136-a56f-93326f08a0ee nodeName:}" failed. No retries permitted until 2026-07-17 18:10:56.722509809 +0000 UTC m=+67.137247815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls") pod "image-registry-86fc465cf-mhntk" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee") : secret "image-registry-tls" not found Jul 17 18:10:48.583327 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:48.583288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:48.585666 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:48.585635 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f6e97-8740-4378-8965-a3d69b83e938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9ngn\" (UID: \"e58f6e97-8740-4378-8965-a3d69b83e938\") " pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:48.612618 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:48.612581 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z9ngn" Jul 17 18:10:48.743278 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:48.743181 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z9ngn"] Jul 17 18:10:48.746028 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:48.745998 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58f6e97_8740_4378_8965_a3d69b83e938.slice/crio-43892c8b9a33ae726359faa71e7119f8f7a5eef6c7f468cbb6a2a9a5f041f2a3 WatchSource:0}: Error finding container 43892c8b9a33ae726359faa71e7119f8f7a5eef6c7f468cbb6a2a9a5f041f2a3: Status 404 returned error can't find the container with id 43892c8b9a33ae726359faa71e7119f8f7a5eef6c7f468cbb6a2a9a5f041f2a3 Jul 17 18:10:49.364545 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:49.364467 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9ngn" event={"ID":"e58f6e97-8740-4378-8965-a3d69b83e938","Type":"ContainerStarted","Data":"43892c8b9a33ae726359faa71e7119f8f7a5eef6c7f468cbb6a2a9a5f041f2a3"} Jul 17 18:10:50.289660 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:50.289624 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txxzb" Jul 17 18:10:54.381097 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:54.381043 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9ngn" event={"ID":"e58f6e97-8740-4378-8965-a3d69b83e938","Type":"ContainerStarted","Data":"3baab5a5ccdcd68bee91f829b6bba858760a1a3e1cb6cd0eac1ff416a8040172"} Jul 17 18:10:55.748557 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.748514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:55.751683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.751658 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:10:55.762284 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.762256 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32593b2d-c97f-4e09-941c-0c2aaa602360-original-pull-secret\") pod \"global-pull-secret-syncer-zv9q4\" (UID: \"32593b2d-c97f-4e09-941c-0c2aaa602360\") " pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:55.849883 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.849841 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:55.852819 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.852792 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:10:55.863606 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.863583 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6198f643-e16b-4409-a9de-a353b91e305c-metrics-certs\") pod \"network-metrics-daemon-xlrp2\" (UID: \"6198f643-e16b-4409-a9de-a353b91e305c\") " pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:55.950283 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.950240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:55.953605 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.953582 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:10:55.964098 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.964071 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:10:55.974225 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.974197 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bw25\" (UniqueName: \"kubernetes.io/projected/6b3b6408-9e70-498c-b543-1fbb4eef10fe-kube-api-access-4bw25\") pod \"network-check-target-gxfcg\" (UID: \"6b3b6408-9e70-498c-b543-1fbb4eef10fe\") " pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:55.982290 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.982269 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47jdr\"" Jul 17 18:10:55.987395 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.987377 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9q4" Jul 17 18:10:55.990512 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.990483 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:10:55.996328 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:55.996300 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nj845\"" Jul 17 18:10:56.004024 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.003947 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlrp2" Jul 17 18:10:56.759032 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.758984 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:56.759463 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.759039 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:56.761700 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.761651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d905cff9-94ae-47e1-9c19-490ebe8a40e3-metrics-tls\") pod \"dns-default-f8dn9\" (UID: \"d905cff9-94ae-47e1-9c19-490ebe8a40e3\") " pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:56.761815 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.761781 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"image-registry-86fc465cf-mhntk\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:56.968171 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.968141 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ksssl\"" Jul 17 18:10:56.975580 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.975553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:56.979163 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.979137 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vkv2q\"" Jul 17 18:10:56.987390 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:56.987354 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f8dn9" Jul 17 18:10:57.073267 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.073234 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh"] Jul 17 18:10:57.109020 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.108992 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x"] Jul 17 18:10:57.109176 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.109087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.112312 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.112294 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 17 18:10:57.112576 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.112560 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 17 18:10:57.112937 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.112918 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 17 18:10:57.113069 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.113058 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 17 18:10:57.131163 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.131133 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86fc465cf-mhntk"] Jul 17 18:10:57.131303 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.131191 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh"] Jul 17 18:10:57.131303 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.131219 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x"] Jul 17 18:10:57.131303 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.131229 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.134004 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.133984 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 17 18:10:57.134004 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.133994 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 17 18:10:57.134157 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.133987 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 17 18:10:57.134220 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.134165 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 17 18:10:57.162912 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.162884 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1217ce4b-0def-469e-b890-4d7065ff853b-klusterlet-config\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.162912 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.162909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxv8\" (UniqueName: \"kubernetes.io/projected/1217ce4b-0def-469e-b890-4d7065ff853b-kube-api-access-hgxv8\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.163105 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.162931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1217ce4b-0def-469e-b890-4d7065ff853b-tmp\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.263480 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-hub\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.263480 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.263480 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxv8\" (UniqueName: \"kubernetes.io/projected/1217ce4b-0def-469e-b890-4d7065ff853b-kube-api-access-hgxv8\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.263729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1217ce4b-0def-469e-b890-4d7065ff853b-tmp\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.263729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.263729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263561 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-ca\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.263729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgkt\" (UniqueName: \"kubernetes.io/projected/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-kube-api-access-8zgkt\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.263729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.263729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.263714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1217ce4b-0def-469e-b890-4d7065ff853b-klusterlet-config\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.264098 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.264022 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1217ce4b-0def-469e-b890-4d7065ff853b-tmp\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.266023 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.265998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1217ce4b-0def-469e-b890-4d7065ff853b-klusterlet-config\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.273607 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.273583 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxv8\" (UniqueName: \"kubernetes.io/projected/1217ce4b-0def-469e-b890-4d7065ff853b-kube-api-access-hgxv8\") pod \"klusterlet-addon-workmgr-fff7db445-xbnmh\" (UID: \"1217ce4b-0def-469e-b890-4d7065ff853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.364988 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.364775 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.364988 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.364826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-ca\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.364988 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.364858 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgkt\" (UniqueName: \"kubernetes.io/projected/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-kube-api-access-8zgkt\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.364988 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.364889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.364988 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.364986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-hub\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.365376 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.365014 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.366712 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.366681 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.368745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.368698 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-ca\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.369245 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.369221 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-hub\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.369710 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.369677 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.373349 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.373298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.384566 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.384504 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgkt\" (UniqueName: \"kubernetes.io/projected/c8ac55f4-1223-4cd9-95fa-23e352dc6de5-kube-api-access-8zgkt\") pod \"cluster-proxy-proxy-agent-77798fff5-82f2x\" (UID: \"c8ac55f4-1223-4cd9-95fa-23e352dc6de5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.418848 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.418742 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:10:57.449735 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.449043 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" Jul 17 18:10:57.510264 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.510172 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f8dn9"] Jul 17 18:10:57.533791 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.533710 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gxfcg"] Jul 17 18:10:57.533791 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:57.533759 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3b6408_9e70_498c_b543_1fbb4eef10fe.slice/crio-4dec6f2f171d6648859895984d757f40291c5dc76ac521b5f6b4948118d0110d WatchSource:0}: Error finding container 4dec6f2f171d6648859895984d757f40291c5dc76ac521b5f6b4948118d0110d: Status 404 returned error can't find the container with id 4dec6f2f171d6648859895984d757f40291c5dc76ac521b5f6b4948118d0110d Jul 17 18:10:57.596885 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.596793 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh"] Jul 17 18:10:57.631664 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.631642 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x"] Jul 17 18:10:57.712196 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:57.712155 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1217ce4b_0def_469e_b890_4d7065ff853b.slice/crio-6d38f7d3fd75925f4f1fd95f9d86603df804f0f74bf5598521754f4f669d5d55 WatchSource:0}: Error finding container 6d38f7d3fd75925f4f1fd95f9d86603df804f0f74bf5598521754f4f669d5d55: Status 404 returned error can't find the container with id 6d38f7d3fd75925f4f1fd95f9d86603df804f0f74bf5598521754f4f669d5d55 Jul 17 18:10:57.712866 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:57.712840 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8ac55f4_1223_4cd9_95fa_23e352dc6de5.slice/crio-697027b2be2e2edcc9fd9dfbf725680992c789f3333fe5e2db34111ba6033a40 WatchSource:0}: Error finding container 697027b2be2e2edcc9fd9dfbf725680992c789f3333fe5e2db34111ba6033a40: Status 404 returned error can't find the container with id 697027b2be2e2edcc9fd9dfbf725680992c789f3333fe5e2db34111ba6033a40 Jul 17 18:10:57.750617 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.750592 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xlrp2"] Jul 17 18:10:57.752752 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:57.752730 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6198f643_e16b_4409_a9de_a353b91e305c.slice/crio-6f68f6c0df92b1281edfc0cace5c85c0d6714ca297aea7afdfd3109df809be55 WatchSource:0}: Error finding container 6f68f6c0df92b1281edfc0cace5c85c0d6714ca297aea7afdfd3109df809be55: Status 404 returned error can't find the container with id 6f68f6c0df92b1281edfc0cace5c85c0d6714ca297aea7afdfd3109df809be55 Jul 17 18:10:57.756411 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.756386 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86fc465cf-mhntk"] Jul 17 18:10:57.757678 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:57.757656 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zv9q4"] Jul 17 18:10:57.758828 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:57.758805 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae35be8c_027a_4136_a56f_93326f08a0ee.slice/crio-26842a61a3e8ee2f2b90e46a16ada893d2172df3c876848d722f7ff008fa2d83 WatchSource:0}: Error finding container 26842a61a3e8ee2f2b90e46a16ada893d2172df3c876848d722f7ff008fa2d83: Status 404 returned error can't find the container with id 26842a61a3e8ee2f2b90e46a16ada893d2172df3c876848d722f7ff008fa2d83 Jul 17 18:10:57.764483 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:10:57.764461 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32593b2d_c97f_4e09_941c_0c2aaa602360.slice/crio-d2870c7b5eaf6d48aa2050c47fac7b509be361fd2a4c534a5e19195b1aabab1d WatchSource:0}: Error finding container d2870c7b5eaf6d48aa2050c47fac7b509be361fd2a4c534a5e19195b1aabab1d: Status 404 returned error can't find the container with id d2870c7b5eaf6d48aa2050c47fac7b509be361fd2a4c534a5e19195b1aabab1d Jul 17 18:10:58.396536 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.396447 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zv9q4" event={"ID":"32593b2d-c97f-4e09-941c-0c2aaa602360","Type":"ContainerStarted","Data":"d2870c7b5eaf6d48aa2050c47fac7b509be361fd2a4c534a5e19195b1aabab1d"} Jul 17 18:10:58.398302 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.398229 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" event={"ID":"c8ac55f4-1223-4cd9-95fa-23e352dc6de5","Type":"ContainerStarted","Data":"697027b2be2e2edcc9fd9dfbf725680992c789f3333fe5e2db34111ba6033a40"} Jul 17 18:10:58.400139 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.400108 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" event={"ID":"1217ce4b-0def-469e-b890-4d7065ff853b","Type":"ContainerStarted","Data":"6d38f7d3fd75925f4f1fd95f9d86603df804f0f74bf5598521754f4f669d5d55"} Jul 17 18:10:58.402755 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.402685 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" event={"ID":"ae35be8c-027a-4136-a56f-93326f08a0ee","Type":"ContainerStarted","Data":"e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019"} Jul 17 18:10:58.402882 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.402766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" event={"ID":"ae35be8c-027a-4136-a56f-93326f08a0ee","Type":"ContainerStarted","Data":"26842a61a3e8ee2f2b90e46a16ada893d2172df3c876848d722f7ff008fa2d83"} Jul 17 18:10:58.403521 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.403064 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:10:58.405595 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.405528 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xlrp2" event={"ID":"6198f643-e16b-4409-a9de-a353b91e305c","Type":"ContainerStarted","Data":"6f68f6c0df92b1281edfc0cace5c85c0d6714ca297aea7afdfd3109df809be55"} Jul 17 18:10:58.410120 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.410043 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9ngn" event={"ID":"e58f6e97-8740-4378-8965-a3d69b83e938","Type":"ContainerStarted","Data":"f9126fd649d76f9a51106106148eabbfc7feb71e5daa73bd56e8636358deaf14"} Jul 17 18:10:58.412017 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.411929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f8dn9" event={"ID":"d905cff9-94ae-47e1-9c19-490ebe8a40e3","Type":"ContainerStarted","Data":"d39656ed7c2b51345118fd32507c3612b62118709b416fd27cbd48b22b9c434f"} Jul 17 18:10:58.415463 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.415397 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gxfcg" event={"ID":"6b3b6408-9e70-498c-b543-1fbb4eef10fe","Type":"ContainerStarted","Data":"4dec6f2f171d6648859895984d757f40291c5dc76ac521b5f6b4948118d0110d"} Jul 17 18:10:58.424562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:10:58.424297 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" podStartSLOduration=55.424277664 podStartE2EDuration="55.424277664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:58.424278676 +0000 UTC m=+68.839016706" watchObservedRunningTime="2026-07-17 18:10:58.424277664 +0000 UTC m=+68.839015669" Jul 17 18:11:09.464416 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.464379 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" event={"ID":"c8ac55f4-1223-4cd9-95fa-23e352dc6de5","Type":"ContainerStarted","Data":"6ef277aeac2038438454d5d89f70d5a5d847e02ae225a25db3098b10b1bde31e"} Jul 17 18:11:09.464934 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.464430 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" event={"ID":"c8ac55f4-1223-4cd9-95fa-23e352dc6de5","Type":"ContainerStarted","Data":"c00d7664e0dc48ab6ec5ca4a29c7ab4465db10c47b5d646beb5704ef98047828"} Jul 17 18:11:09.464934 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.464445 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" event={"ID":"c8ac55f4-1223-4cd9-95fa-23e352dc6de5","Type":"ContainerStarted","Data":"591422e04358613538f6d0185b7181cba43a520965581d6eb30e962a48f10224"} Jul 17 18:11:09.465908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.465878 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" event={"ID":"1217ce4b-0def-469e-b890-4d7065ff853b","Type":"ContainerStarted","Data":"e998a7736b706d345ecf44c9cedd555cd252e9ee0a87ac9064c4470ad0d0d1fb"} Jul 17 18:11:09.466040 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.466018 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:11:09.468034 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.467940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xlrp2" event={"ID":"6198f643-e16b-4409-a9de-a353b91e305c","Type":"ContainerStarted","Data":"661bd48c5d6c68335052f3ad802d5135bd7cc52b2919a6b292a5fb76c4607602"} Jul 17 18:11:09.468034 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.468018 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xlrp2" event={"ID":"6198f643-e16b-4409-a9de-a353b91e305c","Type":"ContainerStarted","Data":"7b41754526ae830375157b9c71336c25187e97f6f89019d94393c2030d86a187"} Jul 17 18:11:09.468219 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.468074 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" Jul 17 18:11:09.470047 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.470018 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9ngn" event={"ID":"e58f6e97-8740-4378-8965-a3d69b83e938","Type":"ContainerStarted","Data":"706ca18f68b3729e8904d537e8edcdd4f3a4a99e46479236110354330e5d82d2"} Jul 17 18:11:09.471803 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.471778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f8dn9" event={"ID":"d905cff9-94ae-47e1-9c19-490ebe8a40e3","Type":"ContainerStarted","Data":"6bc602513377fd48578c8d99b45b054c5a5bf4d93b79511ec113faa7771e0095"} Jul 17 18:11:09.471907 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.471816 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f8dn9" event={"ID":"d905cff9-94ae-47e1-9c19-490ebe8a40e3","Type":"ContainerStarted","Data":"3b22bfd51c45da9c4bb2ed391cdcc2e2e3457f8857d51ce6a6c32f92c432bfd9"} Jul 17 18:11:09.471907 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.471837 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f8dn9" Jul 17 18:11:09.473183 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.473163 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gxfcg" event={"ID":"6b3b6408-9e70-498c-b543-1fbb4eef10fe","Type":"ContainerStarted","Data":"b6c4022cafa5a4df0240095ed482c20b8310a3ee75466d4e636863237079b4e6"} Jul 17 18:11:09.473289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.473212 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:11:09.474638 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.474617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zv9q4" event={"ID":"32593b2d-c97f-4e09-941c-0c2aaa602360","Type":"ContainerStarted","Data":"48304d76ae88fccfa1dfee562aa163b33cd56b2868425a448635fd6023064333"} Jul 17 18:11:09.486809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.486470 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77798fff5-82f2x" podStartSLOduration=1.343820728 podStartE2EDuration="12.486454674s" totalImagesPullingTime="11.142633946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.485882562 +0000 UTC m=+79.900620603" watchObservedRunningTime="2026-07-17 18:11:09.486454674 +0000 UTC m=+79.901192703" Jul 17 18:11:09.506437 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.506391 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z9ngn" podStartSLOduration=17.580707117 podStartE2EDuration="37.50637285s" totalImagesPullingTime="19.925665733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:10:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.506277182 +0000 UTC m=+79.921015275" watchObservedRunningTime="2026-07-17 18:11:09.50637285 +0000 UTC m=+79.921110858" Jul 17 18:11:09.525543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.525497 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fff7db445-xbnmh" podStartSLOduration=1.375937363 podStartE2EDuration="12.525477172s" totalImagesPullingTime="11.149539809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.524867216 +0000 UTC m=+79.939605254" watchObservedRunningTime="2026-07-17 18:11:09.525477172 +0000 UTC m=+79.940215203" Jul 17 18:11:09.542375 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.542334 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xlrp2" podStartSLOduration=68.436401667 podStartE2EDuration="1m19.542322265s" totalImagesPullingTime="11.105920598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.541116512 +0000 UTC m=+79.955854539" watchObservedRunningTime="2026-07-17 18:11:09.542322265 +0000 UTC m=+79.957060289" Jul 17 18:11:09.556943 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.556900 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gxfcg" podStartSLOduration=68.21735081 podStartE2EDuration="1m19.556886695s" totalImagesPullingTime="11.339535885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.555915309 +0000 UTC m=+79.970653361" watchObservedRunningTime="2026-07-17 18:11:09.556886695 +0000 UTC m=+79.971624755" Jul 17 18:11:09.574371 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.574314 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f8dn9" podStartSLOduration=34.230733247 podStartE2EDuration="45.574294968s" totalImagesPullingTime="11.343561721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.572772312 +0000 UTC m=+79.987510339" watchObservedRunningTime="2026-07-17 18:11:09.574294968 +0000 UTC m=+79.989032995" Jul 17 18:11:09.589296 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:09.589237 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zv9q4" podStartSLOduration=68.466590177 podStartE2EDuration="1m19.58921908s" totalImagesPullingTime="11.122628903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.589051628 +0000 UTC m=+80.003789656" watchObservedRunningTime="2026-07-17 18:11:09.58921908 +0000 UTC m=+80.003957108" Jul 17 18:11:18.408664 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:18.408547 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:11:19.480457 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:19.480420 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f8dn9" Jul 17 18:11:22.254418 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.254384 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4wn9w"] Jul 17 18:11:22.478801 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.478773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.481844 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.481818 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 18:11:22.482153 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.482131 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 18:11:22.482229 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.482217 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 18:11:22.482292 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.482268 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 18:11:22.482494 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.482476 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 18:11:22.482494 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.482490 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r66sj\"" Jul 17 18:11:22.482688 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.482493 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 18:11:22.555457 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555371 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-wtmp\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555457 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555436 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-textfile\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-sys\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-tls\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-root\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-accelerators-collector-config\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555683 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555662 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e319103e-7fd0-4971-ac40-4435bc82eb99-metrics-client-ca\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.555993 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.555706 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl96l\" (UniqueName: \"kubernetes.io/projected/e319103e-7fd0-4971-ac40-4435bc82eb99-kube-api-access-pl96l\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656662 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-sys\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656662 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-tls\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-root\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-accelerators-collector-config\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e319103e-7fd0-4971-ac40-4435bc82eb99-metrics-client-ca\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656758 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-sys\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl96l\" (UniqueName: \"kubernetes.io/projected/e319103e-7fd0-4971-ac40-4435bc82eb99-kube-api-access-pl96l\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-root\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-wtmp\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-textfile\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656924 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-wtmp\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.656940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.656928 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.657482 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.657457 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-accelerators-collector-config\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.657630 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.657500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e319103e-7fd0-4971-ac40-4435bc82eb99-metrics-client-ca\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.657630 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.657505 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-textfile\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.659870 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.659848 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.660050 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.660016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e319103e-7fd0-4971-ac40-4435bc82eb99-node-exporter-tls\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.665719 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.665689 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl96l\" (UniqueName: \"kubernetes.io/projected/e319103e-7fd0-4971-ac40-4435bc82eb99-kube-api-access-pl96l\") pod \"node-exporter-4wn9w\" (UID: \"e319103e-7fd0-4971-ac40-4435bc82eb99\") " pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.789813 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:22.789775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4wn9w" Jul 17 18:11:22.840356 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:11:22.840326 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode319103e_7fd0_4971_ac40_4435bc82eb99.slice/crio-2641871f048f166405271ef89669f0a1a8c6dfdf5bead07bebab769a99124a18 WatchSource:0}: Error finding container 2641871f048f166405271ef89669f0a1a8c6dfdf5bead07bebab769a99124a18: Status 404 returned error can't find the container with id 2641871f048f166405271ef89669f0a1a8c6dfdf5bead07bebab769a99124a18 Jul 17 18:11:23.440209 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.440140 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" podUID="ae35be8c-027a-4136-a56f-93326f08a0ee" containerName="registry" containerID="cri-o://e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019" gracePeriod=30 Jul 17 18:11:23.528644 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.528605 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wn9w" event={"ID":"e319103e-7fd0-4971-ac40-4435bc82eb99","Type":"ContainerStarted","Data":"2641871f048f166405271ef89669f0a1a8c6dfdf5bead07bebab769a99124a18"} Jul 17 18:11:23.743599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.743564 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:11:23.867681 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.867651 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-trusted-ca\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.867876 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.867729 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmgls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-kube-api-access-bmgls\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.867876 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.867773 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-bound-sa-token\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.867876 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.867823 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-installation-pull-secrets\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.868072 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.867937 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-certificates\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.868072 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.867998 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.868072 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.868047 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae35be8c-027a-4136-a56f-93326f08a0ee-ca-trust-extracted\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.868224 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.868078 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-image-registry-private-configuration\") pod \"ae35be8c-027a-4136-a56f-93326f08a0ee\" (UID: \"ae35be8c-027a-4136-a56f-93326f08a0ee\") " Jul 17 18:11:23.868909 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.868330 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:11:23.868909 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.868577 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:11:23.871260 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.871210 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:11:23.871493 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.871470 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:11:23.871599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.871578 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:11:23.871873 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.871848 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-kube-api-access-bmgls" (OuterVolumeSpecName: "kube-api-access-bmgls") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "kube-api-access-bmgls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:11:23.873387 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.873355 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:11:23.879153 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.879117 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae35be8c-027a-4136-a56f-93326f08a0ee-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ae35be8c-027a-4136-a56f-93326f08a0ee" (UID: "ae35be8c-027a-4136-a56f-93326f08a0ee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:11:23.969282 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969201 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmgls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-kube-api-access-bmgls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:23.969282 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969235 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-bound-sa-token\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:23.969282 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969247 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-installation-pull-secrets\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:23.969282 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969259 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-certificates\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:23.969282 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969271 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae35be8c-027a-4136-a56f-93326f08a0ee-registry-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:23.969282 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969282 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae35be8c-027a-4136-a56f-93326f08a0ee-ca-trust-extracted\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:23.969595 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969297 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae35be8c-027a-4136-a56f-93326f08a0ee-image-registry-private-configuration\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:23.969595 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:23.969309 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae35be8c-027a-4136-a56f-93326f08a0ee-trusted-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:11:24.534564 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.534528 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wn9w" event={"ID":"e319103e-7fd0-4971-ac40-4435bc82eb99","Type":"ContainerStarted","Data":"31508e8b91ffa28b04c4d3baaf24d17ca89b56222aaaa19f24b87656d21c3563"} Jul 17 18:11:24.535873 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.535848 2570 generic.go:356] "Generic (PLEG): container finished" podID="ae35be8c-027a-4136-a56f-93326f08a0ee" containerID="e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019" exitCode=0 Jul 17 18:11:24.535991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.535893 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" event={"ID":"ae35be8c-027a-4136-a56f-93326f08a0ee","Type":"ContainerDied","Data":"e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019"} Jul 17 18:11:24.535991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.535929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" event={"ID":"ae35be8c-027a-4136-a56f-93326f08a0ee","Type":"ContainerDied","Data":"26842a61a3e8ee2f2b90e46a16ada893d2172df3c876848d722f7ff008fa2d83"} Jul 17 18:11:24.535991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.535927 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86fc465cf-mhntk" Jul 17 18:11:24.535991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.535948 2570 scope.go:117] "RemoveContainer" containerID="e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019" Jul 17 18:11:24.544249 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.544225 2570 scope.go:117] "RemoveContainer" containerID="e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019" Jul 17 18:11:24.544556 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:11:24.544535 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019\": container with ID starting with e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019 not found: ID does not exist" containerID="e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019" Jul 17 18:11:24.544616 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.544569 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019"} err="failed to get container status \"e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019\": rpc error: code = NotFound desc = could not find container \"e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019\": container with ID starting with e94d11fa996edea0ba0ecdc065dee4c426dabebd1007392cc71358eeea446019 not found: ID does not exist" Jul 17 18:11:24.571745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.571703 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86fc465cf-mhntk"] Jul 17 18:11:24.573199 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:24.573173 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86fc465cf-mhntk"] Jul 17 18:11:25.540807 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:25.540773 2570 generic.go:356] "Generic (PLEG): container finished" podID="e319103e-7fd0-4971-ac40-4435bc82eb99" containerID="31508e8b91ffa28b04c4d3baaf24d17ca89b56222aaaa19f24b87656d21c3563" exitCode=0 Jul 17 18:11:25.541297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:25.540854 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wn9w" event={"ID":"e319103e-7fd0-4971-ac40-4435bc82eb99","Type":"ContainerDied","Data":"31508e8b91ffa28b04c4d3baaf24d17ca89b56222aaaa19f24b87656d21c3563"} Jul 17 18:11:26.071160 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:26.071128 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae35be8c-027a-4136-a56f-93326f08a0ee" path="/var/lib/kubelet/pods/ae35be8c-027a-4136-a56f-93326f08a0ee/volumes" Jul 17 18:11:26.547150 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:26.547121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wn9w" event={"ID":"e319103e-7fd0-4971-ac40-4435bc82eb99","Type":"ContainerStarted","Data":"875a8736668706650f6c3baf2e51dbe95e54c611da23bdf47b4374d81e9de8e1"} Jul 17 18:11:26.547527 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:26.547168 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wn9w" event={"ID":"e319103e-7fd0-4971-ac40-4435bc82eb99","Type":"ContainerStarted","Data":"f766d56389c428af6470c84980b1bb0f63b392bb24b666c614be67f55442eecc"} Jul 17 18:11:26.567321 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:26.567271 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4wn9w" podStartSLOduration=2.948792407 podStartE2EDuration="4.567256143s" totalImagesPullingTime="1.513683679s" totalInitContainerRuntime="104.780057ms" isStatefulPod=true podCreationTimestamp="2026-07-17 18:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:26.565995739 +0000 UTC m=+96.980733765" watchObservedRunningTime="2026-07-17 18:11:26.567256143 +0000 UTC m=+96.981994169" Jul 17 18:11:40.482417 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:11:40.480551 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gxfcg" Jul 17 18:13:43.597635 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.597605 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj"] Jul 17 18:13:43.598098 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.597827 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae35be8c-027a-4136-a56f-93326f08a0ee" containerName="registry" Jul 17 18:13:43.598098 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.597836 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae35be8c-027a-4136-a56f-93326f08a0ee" containerName="registry" Jul 17 18:13:43.598098 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.597875 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae35be8c-027a-4136-a56f-93326f08a0ee" containerName="registry" Jul 17 18:13:43.599535 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.599519 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.602224 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.602202 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 18:13:43.602357 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.602231 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 18:13:43.603128 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.603111 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gnkdr\"" Jul 17 18:13:43.609713 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.609692 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj"] Jul 17 18:13:43.731389 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.731353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.731389 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.731396 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdgv\" (UniqueName: \"kubernetes.io/projected/29d47ccd-a806-4d46-9923-1dd2d374aa05-kube-api-access-tsdgv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.731597 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.731491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.832227 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.832187 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.832415 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.832238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.832415 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.832270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdgv\" (UniqueName: \"kubernetes.io/projected/29d47ccd-a806-4d46-9923-1dd2d374aa05-kube-api-access-tsdgv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.832582 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.832560 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.832638 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.832591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.845571 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.845537 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdgv\" (UniqueName: \"kubernetes.io/projected/29d47ccd-a806-4d46-9923-1dd2d374aa05-kube-api-access-tsdgv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:43.909263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:43.909169 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:13:44.037077 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:44.037048 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj"] Jul 17 18:13:44.041292 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:13:44.041255 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d47ccd_a806_4d46_9923_1dd2d374aa05.slice/crio-26fd6634e2c941399add73ba1d7c2c068fa6a3e061162dc74d475c8f2fd0541e WatchSource:0}: Error finding container 26fd6634e2c941399add73ba1d7c2c068fa6a3e061162dc74d475c8f2fd0541e: Status 404 returned error can't find the container with id 26fd6634e2c941399add73ba1d7c2c068fa6a3e061162dc74d475c8f2fd0541e Jul 17 18:13:44.993348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:44.993305 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" event={"ID":"29d47ccd-a806-4d46-9923-1dd2d374aa05","Type":"ContainerStarted","Data":"26fd6634e2c941399add73ba1d7c2c068fa6a3e061162dc74d475c8f2fd0541e"} Jul 17 18:13:52.018850 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:52.018751 2570 generic.go:356] "Generic (PLEG): container finished" podID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerID="9fa211945f226eb0fad0891a39281d071836154b542573dc5dd741b848e78a76" exitCode=0 Jul 17 18:13:52.019320 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:52.018841 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" event={"ID":"29d47ccd-a806-4d46-9923-1dd2d374aa05","Type":"ContainerDied","Data":"9fa211945f226eb0fad0891a39281d071836154b542573dc5dd741b848e78a76"} Jul 17 18:13:55.032043 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:55.032010 2570 generic.go:356] "Generic (PLEG): container finished" podID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerID="dc9c557815661807e9a2cc1d44c8c018706ed533cbd20150dda549695344ae7e" exitCode=0 Jul 17 18:13:55.032506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:13:55.032057 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" event={"ID":"29d47ccd-a806-4d46-9923-1dd2d374aa05","Type":"ContainerDied","Data":"dc9c557815661807e9a2cc1d44c8c018706ed533cbd20150dda549695344ae7e"} Jul 17 18:14:03.063099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:03.063017 2570 generic.go:356] "Generic (PLEG): container finished" podID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerID="ff1c33a3781a1cbaae531e6ee148237819b6df285bd7574296be9170ae4017d0" exitCode=0 Jul 17 18:14:03.063503 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:03.063109 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" event={"ID":"29d47ccd-a806-4d46-9923-1dd2d374aa05","Type":"ContainerDied","Data":"ff1c33a3781a1cbaae531e6ee148237819b6df285bd7574296be9170ae4017d0"} Jul 17 18:14:04.184904 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.184881 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:14:04.186820 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.186803 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-util\") pod \"29d47ccd-a806-4d46-9923-1dd2d374aa05\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " Jul 17 18:14:04.186879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.186833 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-bundle\") pod \"29d47ccd-a806-4d46-9923-1dd2d374aa05\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " Jul 17 18:14:04.186879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.186863 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsdgv\" (UniqueName: \"kubernetes.io/projected/29d47ccd-a806-4d46-9923-1dd2d374aa05-kube-api-access-tsdgv\") pod \"29d47ccd-a806-4d46-9923-1dd2d374aa05\" (UID: \"29d47ccd-a806-4d46-9923-1dd2d374aa05\") " Jul 17 18:14:04.187457 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.187435 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-bundle" (OuterVolumeSpecName: "bundle") pod "29d47ccd-a806-4d46-9923-1dd2d374aa05" (UID: "29d47ccd-a806-4d46-9923-1dd2d374aa05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:04.189012 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.188979 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d47ccd-a806-4d46-9923-1dd2d374aa05-kube-api-access-tsdgv" (OuterVolumeSpecName: "kube-api-access-tsdgv") pod "29d47ccd-a806-4d46-9923-1dd2d374aa05" (UID: "29d47ccd-a806-4d46-9923-1dd2d374aa05"). InnerVolumeSpecName "kube-api-access-tsdgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:04.191429 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.191409 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-util" (OuterVolumeSpecName: "util") pod "29d47ccd-a806-4d46-9923-1dd2d374aa05" (UID: "29d47ccd-a806-4d46-9923-1dd2d374aa05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:04.287851 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.287819 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdgv\" (UniqueName: \"kubernetes.io/projected/29d47ccd-a806-4d46-9923-1dd2d374aa05-kube-api-access-tsdgv\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:04.287851 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.287848 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-util\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:04.287851 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:04.287860 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29d47ccd-a806-4d46-9923-1dd2d374aa05-bundle\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:05.071668 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:05.071639 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" Jul 17 18:14:05.071863 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:05.071638 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vkckj" event={"ID":"29d47ccd-a806-4d46-9923-1dd2d374aa05","Type":"ContainerDied","Data":"26fd6634e2c941399add73ba1d7c2c068fa6a3e061162dc74d475c8f2fd0541e"} Jul 17 18:14:05.071863 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:05.071756 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26fd6634e2c941399add73ba1d7c2c068fa6a3e061162dc74d475c8f2fd0541e" Jul 17 18:14:11.541109 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541030 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr"] Jul 17 18:14:11.541543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541243 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerName="util" Jul 17 18:14:11.541543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541254 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerName="util" Jul 17 18:14:11.541543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541269 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerName="pull" Jul 17 18:14:11.541543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541274 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerName="pull" Jul 17 18:14:11.541543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541282 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerName="extract" Jul 17 18:14:11.541543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541288 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerName="extract" Jul 17 18:14:11.541543 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.541330 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29d47ccd-a806-4d46-9923-1dd2d374aa05" containerName="extract" Jul 17 18:14:11.543884 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.543869 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.546453 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.546425 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xf86q\"" Jul 17 18:14:11.546453 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.546425 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 17 18:14:11.546668 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.546428 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:14:11.558402 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.558371 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr"] Jul 17 18:14:11.634675 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.634639 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce642c05-0e68-4df8-9082-7a231cb7b666-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jjkr\" (UID: \"ce642c05-0e68-4df8-9082-7a231cb7b666\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.634852 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.634682 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8wd\" (UniqueName: \"kubernetes.io/projected/ce642c05-0e68-4df8-9082-7a231cb7b666-kube-api-access-5d8wd\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jjkr\" (UID: \"ce642c05-0e68-4df8-9082-7a231cb7b666\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.735077 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.735037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce642c05-0e68-4df8-9082-7a231cb7b666-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jjkr\" (UID: \"ce642c05-0e68-4df8-9082-7a231cb7b666\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.735228 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.735094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8wd\" (UniqueName: \"kubernetes.io/projected/ce642c05-0e68-4df8-9082-7a231cb7b666-kube-api-access-5d8wd\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jjkr\" (UID: \"ce642c05-0e68-4df8-9082-7a231cb7b666\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.735522 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.735506 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce642c05-0e68-4df8-9082-7a231cb7b666-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jjkr\" (UID: \"ce642c05-0e68-4df8-9082-7a231cb7b666\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.746030 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.745995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8wd\" (UniqueName: \"kubernetes.io/projected/ce642c05-0e68-4df8-9082-7a231cb7b666-kube-api-access-5d8wd\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jjkr\" (UID: \"ce642c05-0e68-4df8-9082-7a231cb7b666\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.852739 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.852649 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" Jul 17 18:14:11.982742 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:11.982713 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr"] Jul 17 18:14:11.986635 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:14:11.986610 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce642c05_0e68_4df8_9082_7a231cb7b666.slice/crio-56b6e6a3c3fb84176306eda162bd94ca85e6e5b9bd621388564fc25c04744c2f WatchSource:0}: Error finding container 56b6e6a3c3fb84176306eda162bd94ca85e6e5b9bd621388564fc25c04744c2f: Status 404 returned error can't find the container with id 56b6e6a3c3fb84176306eda162bd94ca85e6e5b9bd621388564fc25c04744c2f Jul 17 18:14:12.094660 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:12.094628 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" event={"ID":"ce642c05-0e68-4df8-9082-7a231cb7b666","Type":"ContainerStarted","Data":"56b6e6a3c3fb84176306eda162bd94ca85e6e5b9bd621388564fc25c04744c2f"} Jul 17 18:14:14.104078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:14.103989 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" event={"ID":"ce642c05-0e68-4df8-9082-7a231cb7b666","Type":"ContainerStarted","Data":"a120020cfc71cfbb1dd38a86a0bf808f9467b1b0657be17c75f355d39d77d170"} Jul 17 18:14:14.128324 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:14.128273 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jjkr" podStartSLOduration=1.2642811219999999 podStartE2EDuration="3.128252513s" totalImagesPullingTime="1.863971391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:14.126215023 +0000 UTC m=+264.540953052" watchObservedRunningTime="2026-07-17 18:14:14.128252513 +0000 UTC m=+264.542990540" Jul 17 18:14:15.492371 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.492338 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z"] Jul 17 18:14:15.495824 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.495800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.499406 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.499383 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 18:14:15.502186 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.502164 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 18:14:15.502296 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.502186 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gnkdr\"" Jul 17 18:14:15.509036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.509011 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z"] Jul 17 18:14:15.560078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.560047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.560253 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.560087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.560253 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.560137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/7cc6d9ea-2514-460f-b3d1-4382921b0e09-kube-api-access-f4q4t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.661314 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.661281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.661517 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.661331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/7cc6d9ea-2514-460f-b3d1-4382921b0e09-kube-api-access-f4q4t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.661517 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.661385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.661738 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.661714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.661738 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.661736 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.670438 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.670408 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/7cc6d9ea-2514-460f-b3d1-4382921b0e09-kube-api-access-f4q4t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.804673 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.804593 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:15.937567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:15.937543 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z"] Jul 17 18:14:15.939243 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:14:15.939208 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc6d9ea_2514_460f_b3d1_4382921b0e09.slice/crio-4b51c487fb2bd37de6f54b18062be6edf9012d4bb0489822ebf382b3ea52569b WatchSource:0}: Error finding container 4b51c487fb2bd37de6f54b18062be6edf9012d4bb0489822ebf382b3ea52569b: Status 404 returned error can't find the container with id 4b51c487fb2bd37de6f54b18062be6edf9012d4bb0489822ebf382b3ea52569b Jul 17 18:14:16.112605 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:16.112514 2570 generic.go:356] "Generic (PLEG): container finished" podID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerID="127ec9449f1879d742196c8742b0fd38aad95149397319746cac7109ca8b8424" exitCode=0 Jul 17 18:14:16.112605 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:16.112586 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" event={"ID":"7cc6d9ea-2514-460f-b3d1-4382921b0e09","Type":"ContainerDied","Data":"127ec9449f1879d742196c8742b0fd38aad95149397319746cac7109ca8b8424"} Jul 17 18:14:16.112807 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:16.112627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" event={"ID":"7cc6d9ea-2514-460f-b3d1-4382921b0e09","Type":"ContainerStarted","Data":"4b51c487fb2bd37de6f54b18062be6edf9012d4bb0489822ebf382b3ea52569b"} Jul 17 18:14:17.364862 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.364827 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj"] Jul 17 18:14:17.372939 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.372910 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:17.375659 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.375631 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 18:14:17.376489 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.376462 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qf99f\"" Jul 17 18:14:17.376640 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.376463 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 18:14:17.378824 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.378804 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj"] Jul 17 18:14:17.474690 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.474643 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gzmjj\" (UID: \"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:17.474690 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.474692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59d69\" (UniqueName: \"kubernetes.io/projected/c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc-kube-api-access-59d69\") pod \"cert-manager-cainjector-559cdd8c87-gzmjj\" (UID: \"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:17.575249 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.575203 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gzmjj\" (UID: \"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:17.575454 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.575281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59d69\" (UniqueName: \"kubernetes.io/projected/c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc-kube-api-access-59d69\") pod \"cert-manager-cainjector-559cdd8c87-gzmjj\" (UID: \"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:17.586819 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.586783 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gzmjj\" (UID: \"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:17.586997 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.586926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59d69\" (UniqueName: \"kubernetes.io/projected/c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc-kube-api-access-59d69\") pod \"cert-manager-cainjector-559cdd8c87-gzmjj\" (UID: \"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:17.689700 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:17.689659 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" Jul 17 18:14:18.022496 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:18.022468 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj"] Jul 17 18:14:18.025915 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:14:18.025890 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9027cc6_ab2e_4d2f_a5a7_a648e0f8fdcc.slice/crio-d258189028f27633d96bdeb154cc51f72730c056b1a485eff795e400fc86189a WatchSource:0}: Error finding container d258189028f27633d96bdeb154cc51f72730c056b1a485eff795e400fc86189a: Status 404 returned error can't find the container with id d258189028f27633d96bdeb154cc51f72730c056b1a485eff795e400fc86189a Jul 17 18:14:18.122352 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:18.122314 2570 generic.go:356] "Generic (PLEG): container finished" podID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerID="7a94702aaca8b048fc311f7eeb26832c85ef68e804cd463365b7bf8985906bdd" exitCode=0 Jul 17 18:14:18.122534 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:18.122402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" event={"ID":"7cc6d9ea-2514-460f-b3d1-4382921b0e09","Type":"ContainerDied","Data":"7a94702aaca8b048fc311f7eeb26832c85ef68e804cd463365b7bf8985906bdd"} Jul 17 18:14:18.123497 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:18.123477 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" event={"ID":"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc","Type":"ContainerStarted","Data":"d258189028f27633d96bdeb154cc51f72730c056b1a485eff795e400fc86189a"} Jul 17 18:14:19.129783 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:19.129745 2570 generic.go:356] "Generic (PLEG): container finished" podID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerID="6f2192d22cff5b60d3b43e96f8a5adb9562f7d04593063672300fc3bd3d2311e" exitCode=0 Jul 17 18:14:19.130257 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:19.129811 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" event={"ID":"7cc6d9ea-2514-460f-b3d1-4382921b0e09","Type":"ContainerDied","Data":"6f2192d22cff5b60d3b43e96f8a5adb9562f7d04593063672300fc3bd3d2311e"} Jul 17 18:14:20.715157 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.715130 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:20.796051 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.795866 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-util\") pod \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " Jul 17 18:14:20.796051 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.795922 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-bundle\") pod \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " Jul 17 18:14:20.796500 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.796437 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-bundle" (OuterVolumeSpecName: "bundle") pod "7cc6d9ea-2514-460f-b3d1-4382921b0e09" (UID: "7cc6d9ea-2514-460f-b3d1-4382921b0e09"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:20.801646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.801619 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-util" (OuterVolumeSpecName: "util") pod "7cc6d9ea-2514-460f-b3d1-4382921b0e09" (UID: "7cc6d9ea-2514-460f-b3d1-4382921b0e09"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:20.896936 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.896872 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/7cc6d9ea-2514-460f-b3d1-4382921b0e09-kube-api-access-f4q4t\") pod \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\" (UID: \"7cc6d9ea-2514-460f-b3d1-4382921b0e09\") " Jul 17 18:14:20.897144 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.897080 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-util\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:20.897144 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.897098 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cc6d9ea-2514-460f-b3d1-4382921b0e09-bundle\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:20.899176 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.899140 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc6d9ea-2514-460f-b3d1-4382921b0e09-kube-api-access-f4q4t" (OuterVolumeSpecName: "kube-api-access-f4q4t") pod "7cc6d9ea-2514-460f-b3d1-4382921b0e09" (UID: "7cc6d9ea-2514-460f-b3d1-4382921b0e09"). InnerVolumeSpecName "kube-api-access-f4q4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:20.997696 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:20.997611 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/7cc6d9ea-2514-460f-b3d1-4382921b0e09-kube-api-access-f4q4t\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.143745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:21.143705 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" event={"ID":"c9027cc6-ab2e-4d2f-a5a7-a648e0f8fdcc","Type":"ContainerStarted","Data":"051bf34f6acff7b09899b0132f57a2798967aa12837f6e434ec9d06d5f6af4e1"} Jul 17 18:14:21.145565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:21.145541 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" event={"ID":"7cc6d9ea-2514-460f-b3d1-4382921b0e09","Type":"ContainerDied","Data":"4b51c487fb2bd37de6f54b18062be6edf9012d4bb0489822ebf382b3ea52569b"} Jul 17 18:14:21.145565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:21.145561 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdxb6z" Jul 17 18:14:21.145732 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:21.145612 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b51c487fb2bd37de6f54b18062be6edf9012d4bb0489822ebf382b3ea52569b" Jul 17 18:14:21.795022 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:21.794950 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gzmjj" podStartSLOduration=2.063887457 podStartE2EDuration="4.794929972s" totalImagesPullingTime="2.731042515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:21.160082356 +0000 UTC m=+271.574820384" watchObservedRunningTime="2026-07-17 18:14:21.794929972 +0000 UTC m=+272.209667999" Jul 17 18:14:25.200646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200615 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pz2q9"] Jul 17 18:14:25.201033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200835 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerName="util" Jul 17 18:14:25.201033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200846 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerName="util" Jul 17 18:14:25.201033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200852 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerName="extract" Jul 17 18:14:25.201033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200858 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerName="extract" Jul 17 18:14:25.201033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200877 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerName="pull" Jul 17 18:14:25.201033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200882 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerName="pull" Jul 17 18:14:25.201033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.200917 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc6d9ea-2514-460f-b3d1-4382921b0e09" containerName="extract" Jul 17 18:14:25.203818 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.203803 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.206610 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.206590 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-w6qpk\"" Jul 17 18:14:25.214385 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.214360 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pz2q9"] Jul 17 18:14:25.223394 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.223373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed5a6364-c674-479e-a10c-d849224b3381-bound-sa-token\") pod \"cert-manager-55b4d954d5-pz2q9\" (UID: \"ed5a6364-c674-479e-a10c-d849224b3381\") " pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.223506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.223405 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5jc\" (UniqueName: \"kubernetes.io/projected/ed5a6364-c674-479e-a10c-d849224b3381-kube-api-access-xg5jc\") pod \"cert-manager-55b4d954d5-pz2q9\" (UID: \"ed5a6364-c674-479e-a10c-d849224b3381\") " pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.324338 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.324299 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed5a6364-c674-479e-a10c-d849224b3381-bound-sa-token\") pod \"cert-manager-55b4d954d5-pz2q9\" (UID: \"ed5a6364-c674-479e-a10c-d849224b3381\") " pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.324338 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.324344 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5jc\" (UniqueName: \"kubernetes.io/projected/ed5a6364-c674-479e-a10c-d849224b3381-kube-api-access-xg5jc\") pod \"cert-manager-55b4d954d5-pz2q9\" (UID: \"ed5a6364-c674-479e-a10c-d849224b3381\") " pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.339045 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.339011 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed5a6364-c674-479e-a10c-d849224b3381-bound-sa-token\") pod \"cert-manager-55b4d954d5-pz2q9\" (UID: \"ed5a6364-c674-479e-a10c-d849224b3381\") " pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.339228 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.339209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5jc\" (UniqueName: \"kubernetes.io/projected/ed5a6364-c674-479e-a10c-d849224b3381-kube-api-access-xg5jc\") pod \"cert-manager-55b4d954d5-pz2q9\" (UID: \"ed5a6364-c674-479e-a10c-d849224b3381\") " pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.512533 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.512450 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pz2q9" Jul 17 18:14:25.639486 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:25.639461 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pz2q9"] Jul 17 18:14:25.641583 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:14:25.641555 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded5a6364_c674_479e_a10c_d849224b3381.slice/crio-b43ca9d2d93995cf5feafc03f08ad31af879caae723a51cda0c3980bbb71e2f0 WatchSource:0}: Error finding container b43ca9d2d93995cf5feafc03f08ad31af879caae723a51cda0c3980bbb71e2f0: Status 404 returned error can't find the container with id b43ca9d2d93995cf5feafc03f08ad31af879caae723a51cda0c3980bbb71e2f0 Jul 17 18:14:26.166443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:26.166409 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pz2q9" event={"ID":"ed5a6364-c674-479e-a10c-d849224b3381","Type":"ContainerStarted","Data":"e951a85f33b70ec39f0c8ab8ac77a23305dddcc3307d1c464aae5a60c6b010e2"} Jul 17 18:14:26.166443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:26.166450 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pz2q9" event={"ID":"ed5a6364-c674-479e-a10c-d849224b3381","Type":"ContainerStarted","Data":"b43ca9d2d93995cf5feafc03f08ad31af879caae723a51cda0c3980bbb71e2f0"} Jul 17 18:14:26.185322 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:26.185274 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pz2q9" podStartSLOduration=1.185258277 podStartE2EDuration="1.185258277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:26.18425999 +0000 UTC m=+276.598998017" watchObservedRunningTime="2026-07-17 18:14:26.185258277 +0000 UTC m=+276.599996303" Jul 17 18:14:28.283922 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.283891 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch"] Jul 17 18:14:28.286889 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.286871 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.289473 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.289451 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 18:14:28.289590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.289524 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:14:28.290428 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.290413 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fpqqv\"" Jul 17 18:14:28.295373 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.295351 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch"] Jul 17 18:14:28.348462 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.348423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qkc\" (UniqueName: \"kubernetes.io/projected/370c3cc5-e768-4c2d-b49c-6e7632e35286-kube-api-access-z5qkc\") pod \"openshift-lws-operator-fd8ccff4c-48fch\" (UID: \"370c3cc5-e768-4c2d-b49c-6e7632e35286\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.348462 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.348463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/370c3cc5-e768-4c2d-b49c-6e7632e35286-tmp\") pod \"openshift-lws-operator-fd8ccff4c-48fch\" (UID: \"370c3cc5-e768-4c2d-b49c-6e7632e35286\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.449546 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.449510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qkc\" (UniqueName: \"kubernetes.io/projected/370c3cc5-e768-4c2d-b49c-6e7632e35286-kube-api-access-z5qkc\") pod \"openshift-lws-operator-fd8ccff4c-48fch\" (UID: \"370c3cc5-e768-4c2d-b49c-6e7632e35286\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.449546 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.449551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/370c3cc5-e768-4c2d-b49c-6e7632e35286-tmp\") pod \"openshift-lws-operator-fd8ccff4c-48fch\" (UID: \"370c3cc5-e768-4c2d-b49c-6e7632e35286\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.449935 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.449919 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/370c3cc5-e768-4c2d-b49c-6e7632e35286-tmp\") pod \"openshift-lws-operator-fd8ccff4c-48fch\" (UID: \"370c3cc5-e768-4c2d-b49c-6e7632e35286\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.460152 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.460136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qkc\" (UniqueName: \"kubernetes.io/projected/370c3cc5-e768-4c2d-b49c-6e7632e35286-kube-api-access-z5qkc\") pod \"openshift-lws-operator-fd8ccff4c-48fch\" (UID: \"370c3cc5-e768-4c2d-b49c-6e7632e35286\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.596796 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.596701 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" Jul 17 18:14:28.725496 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:28.725337 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch"] Jul 17 18:14:28.728040 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:14:28.728008 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod370c3cc5_e768_4c2d_b49c_6e7632e35286.slice/crio-bb99da1869b147c10bb5c386c036ecece7b55e59efd33002fad1a1dc1ae78c19 WatchSource:0}: Error finding container bb99da1869b147c10bb5c386c036ecece7b55e59efd33002fad1a1dc1ae78c19: Status 404 returned error can't find the container with id bb99da1869b147c10bb5c386c036ecece7b55e59efd33002fad1a1dc1ae78c19 Jul 17 18:14:29.180356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:29.180319 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" event={"ID":"370c3cc5-e768-4c2d-b49c-6e7632e35286","Type":"ContainerStarted","Data":"bb99da1869b147c10bb5c386c036ecece7b55e59efd33002fad1a1dc1ae78c19"} Jul 17 18:14:31.192352 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:31.192320 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" event={"ID":"370c3cc5-e768-4c2d-b49c-6e7632e35286","Type":"ContainerStarted","Data":"061e8fcdf458ed497d7b86962481316691d6ce5aa0a390405a5e9bff7f91ed86"} Jul 17 18:14:31.207567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:31.207509 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-48fch" podStartSLOduration=1.029193673 podStartE2EDuration="3.207493353s" totalImagesPullingTime="2.17829968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:31.207061357 +0000 UTC m=+281.621799384" watchObservedRunningTime="2026-07-17 18:14:31.207493353 +0000 UTC m=+281.622231379" Jul 17 18:14:36.133601 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.133566 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t"] Jul 17 18:14:36.155094 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.155064 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t"] Jul 17 18:14:36.155225 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.155147 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.157915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.157883 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gnkdr\"" Jul 17 18:14:36.158057 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.157995 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 18:14:36.158057 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.158003 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 18:14:36.208648 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.208622 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.208836 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.208674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97f6p\" (UniqueName: \"kubernetes.io/projected/43484e4c-4b21-44ff-b50c-b2cd87a99cda-kube-api-access-97f6p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.208836 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.208705 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.315844 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.315364 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.316133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.315810 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.316340 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.316321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97f6p\" (UniqueName: \"kubernetes.io/projected/43484e4c-4b21-44ff-b50c-b2cd87a99cda-kube-api-access-97f6p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.316492 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.316478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.316846 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.316818 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.325341 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.325301 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97f6p\" (UniqueName: \"kubernetes.io/projected/43484e4c-4b21-44ff-b50c-b2cd87a99cda-kube-api-access-97f6p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.464054 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.464007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:36.616722 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:36.616697 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t"] Jul 17 18:14:36.618879 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:14:36.618841 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43484e4c_4b21_44ff_b50c_b2cd87a99cda.slice/crio-2b99c1a8f2f748cf15eb64f2265cb21a4e19cc45b68fb127a4b33bacb2a24d5c WatchSource:0}: Error finding container 2b99c1a8f2f748cf15eb64f2265cb21a4e19cc45b68fb127a4b33bacb2a24d5c: Status 404 returned error can't find the container with id 2b99c1a8f2f748cf15eb64f2265cb21a4e19cc45b68fb127a4b33bacb2a24d5c Jul 17 18:14:37.216218 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:37.216182 2570 generic.go:356] "Generic (PLEG): container finished" podID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerID="dee48e27a1cd304d709eacb9dd449d24bb8fa63b1df609af1120881e964b3efa" exitCode=0 Jul 17 18:14:37.216644 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:37.216263 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" event={"ID":"43484e4c-4b21-44ff-b50c-b2cd87a99cda","Type":"ContainerDied","Data":"dee48e27a1cd304d709eacb9dd449d24bb8fa63b1df609af1120881e964b3efa"} Jul 17 18:14:37.216644 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:37.216308 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" event={"ID":"43484e4c-4b21-44ff-b50c-b2cd87a99cda","Type":"ContainerStarted","Data":"2b99c1a8f2f748cf15eb64f2265cb21a4e19cc45b68fb127a4b33bacb2a24d5c"} Jul 17 18:14:38.223376 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:38.223341 2570 generic.go:356] "Generic (PLEG): container finished" podID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerID="a7cedca95cb3e8889e1a4f40c1eaea24ed35697cc78e7311d197b09dbccaff3e" exitCode=0 Jul 17 18:14:38.223792 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:38.223425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" event={"ID":"43484e4c-4b21-44ff-b50c-b2cd87a99cda","Type":"ContainerDied","Data":"a7cedca95cb3e8889e1a4f40c1eaea24ed35697cc78e7311d197b09dbccaff3e"} Jul 17 18:14:39.229753 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:39.229717 2570 generic.go:356] "Generic (PLEG): container finished" podID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerID="e9235f4b4b53c306ba29d530e378fc43ac0875acc7ae3d6db70537c389d145f2" exitCode=0 Jul 17 18:14:39.229753 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:39.229756 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" event={"ID":"43484e4c-4b21-44ff-b50c-b2cd87a99cda","Type":"ContainerDied","Data":"e9235f4b4b53c306ba29d530e378fc43ac0875acc7ae3d6db70537c389d145f2"} Jul 17 18:14:40.354350 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.354326 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:40.449272 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.449241 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97f6p\" (UniqueName: \"kubernetes.io/projected/43484e4c-4b21-44ff-b50c-b2cd87a99cda-kube-api-access-97f6p\") pod \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " Jul 17 18:14:40.449467 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.449297 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-util\") pod \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " Jul 17 18:14:40.449467 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.449313 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-bundle\") pod \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\" (UID: \"43484e4c-4b21-44ff-b50c-b2cd87a99cda\") " Jul 17 18:14:40.450078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.450053 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-bundle" (OuterVolumeSpecName: "bundle") pod "43484e4c-4b21-44ff-b50c-b2cd87a99cda" (UID: "43484e4c-4b21-44ff-b50c-b2cd87a99cda"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:40.451343 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.451322 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43484e4c-4b21-44ff-b50c-b2cd87a99cda-kube-api-access-97f6p" (OuterVolumeSpecName: "kube-api-access-97f6p") pod "43484e4c-4b21-44ff-b50c-b2cd87a99cda" (UID: "43484e4c-4b21-44ff-b50c-b2cd87a99cda"). InnerVolumeSpecName "kube-api-access-97f6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:40.454390 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.454352 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-util" (OuterVolumeSpecName: "util") pod "43484e4c-4b21-44ff-b50c-b2cd87a99cda" (UID: "43484e4c-4b21-44ff-b50c-b2cd87a99cda"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:40.550797 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.550706 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-util\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:40.550797 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.550760 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43484e4c-4b21-44ff-b50c-b2cd87a99cda-bundle\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:40.550797 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:40.550771 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97f6p\" (UniqueName: \"kubernetes.io/projected/43484e4c-4b21-44ff-b50c-b2cd87a99cda-kube-api-access-97f6p\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:14:41.239344 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:41.239313 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" Jul 17 18:14:41.239514 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:41.239315 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpf65t" event={"ID":"43484e4c-4b21-44ff-b50c-b2cd87a99cda","Type":"ContainerDied","Data":"2b99c1a8f2f748cf15eb64f2265cb21a4e19cc45b68fb127a4b33bacb2a24d5c"} Jul 17 18:14:41.239514 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:41.239435 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b99c1a8f2f748cf15eb64f2265cb21a4e19cc45b68fb127a4b33bacb2a24d5c" Jul 17 18:14:49.541560 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541519 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6"] Jul 17 18:14:49.541972 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541782 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerName="util" Jul 17 18:14:49.541972 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541794 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerName="util" Jul 17 18:14:49.541972 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541807 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerName="extract" Jul 17 18:14:49.541972 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541814 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerName="extract" Jul 17 18:14:49.541972 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541823 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerName="pull" Jul 17 18:14:49.541972 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541829 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerName="pull" Jul 17 18:14:49.541972 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.541867 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="43484e4c-4b21-44ff-b50c-b2cd87a99cda" containerName="extract" Jul 17 18:14:49.547197 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.547175 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.550086 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.550059 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 18:14:49.550229 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.550089 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 18:14:49.550229 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.550060 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 18:14:49.550513 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.550497 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 18:14:49.550756 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.550745 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z9sgw\"" Jul 17 18:14:49.565412 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.565380 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6"] Jul 17 18:14:49.619720 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.619678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/54850ee9-6cb1-4e0e-9343-15092fc4581a-manifests\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.619720 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.619718 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54850ee9-6cb1-4e0e-9343-15092fc4581a-tmp\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.619950 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.619754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/54850ee9-6cb1-4e0e-9343-15092fc4581a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.619950 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.619831 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54850ee9-6cb1-4e0e-9343-15092fc4581a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.619950 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.619865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshkh\" (UniqueName: \"kubernetes.io/projected/54850ee9-6cb1-4e0e-9343-15092fc4581a-kube-api-access-vshkh\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.721106 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.721061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54850ee9-6cb1-4e0e-9343-15092fc4581a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.721106 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.721102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshkh\" (UniqueName: \"kubernetes.io/projected/54850ee9-6cb1-4e0e-9343-15092fc4581a-kube-api-access-vshkh\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.721368 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.721134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/54850ee9-6cb1-4e0e-9343-15092fc4581a-manifests\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.721368 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.721159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54850ee9-6cb1-4e0e-9343-15092fc4581a-tmp\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.721368 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.721197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/54850ee9-6cb1-4e0e-9343-15092fc4581a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.721654 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.721629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/54850ee9-6cb1-4e0e-9343-15092fc4581a-manifests\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.723390 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.723362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54850ee9-6cb1-4e0e-9343-15092fc4581a-tmp\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.723596 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.723580 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54850ee9-6cb1-4e0e-9343-15092fc4581a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.723639 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.723586 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/54850ee9-6cb1-4e0e-9343-15092fc4581a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.729375 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.729347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshkh\" (UniqueName: \"kubernetes.io/projected/54850ee9-6cb1-4e0e-9343-15092fc4581a-kube-api-access-vshkh\") pod \"opendatahub-operator-controller-manager-9846c54f8-2qtd6\" (UID: \"54850ee9-6cb1-4e0e-9343-15092fc4581a\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.857274 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.857172 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:49.950881 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.950856 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:14:49.954385 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.954350 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 18:14:49.961991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.961946 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:14:49.998458 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:49.998425 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6"] Jul 17 18:14:50.276500 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:50.276468 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" event={"ID":"54850ee9-6cb1-4e0e-9343-15092fc4581a","Type":"ContainerStarted","Data":"075b2db28c163260f2c151906609ff84fd95c5634974df4f524706fe1380a1ed"} Jul 17 18:14:54.296721 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:54.296683 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" event={"ID":"54850ee9-6cb1-4e0e-9343-15092fc4581a","Type":"ContainerStarted","Data":"34e4f8defc5baba99a05691db523f6bb6fea7ee9e34ae6fe89863dd24006d142"} Jul 17 18:14:55.302245 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:55.302210 2570 generic.go:356] "Generic (PLEG): container finished" podID="54850ee9-6cb1-4e0e-9343-15092fc4581a" containerID="34e4f8defc5baba99a05691db523f6bb6fea7ee9e34ae6fe89863dd24006d142" exitCode=0 Jul 17 18:14:55.302634 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:55.302306 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" event={"ID":"54850ee9-6cb1-4e0e-9343-15092fc4581a","Type":"ContainerDied","Data":"34e4f8defc5baba99a05691db523f6bb6fea7ee9e34ae6fe89863dd24006d142"} Jul 17 18:14:55.302734 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:55.302720 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:14:56.308418 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:56.308379 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" event={"ID":"54850ee9-6cb1-4e0e-9343-15092fc4581a","Type":"ContainerStarted","Data":"73ccd1f5b7749276e6bc5207bfece4ed289bbfc7d43d15966a9539609ab1e098"} Jul 17 18:14:56.308808 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:56.308502 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:14:56.329356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:14:56.329294 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" podStartSLOduration=2.107087486 podStartE2EDuration="7.329273852s" totalImagesPullingTime="4.148449834s" totalInitContainerRuntime="1.073736532s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:56.329126612 +0000 UTC m=+306.743864662" watchObservedRunningTime="2026-07-17 18:14:56.329273852 +0000 UTC m=+306.744011881" Jul 17 18:15:07.314864 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:07.314834 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-2qtd6" Jul 17 18:15:09.692730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.692691 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs"] Jul 17 18:15:09.701263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.701234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.706801 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.706749 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 18:15:09.706969 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.706872 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 18:15:09.707242 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.707225 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 18:15:09.708443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.708426 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tz8nf\"" Jul 17 18:15:09.708532 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.708473 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 18:15:09.722581 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.722546 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs"] Jul 17 18:15:09.777089 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.777009 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2bd523b-e875-4aea-b1f4-def4f0875183-tls-certs\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.777291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.777166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2bd523b-e875-4aea-b1f4-def4f0875183-tmp\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.777291 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.777228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjd7v\" (UniqueName: \"kubernetes.io/projected/a2bd523b-e875-4aea-b1f4-def4f0875183-kube-api-access-gjd7v\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.877672 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.877629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2bd523b-e875-4aea-b1f4-def4f0875183-tls-certs\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.877865 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.877683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2bd523b-e875-4aea-b1f4-def4f0875183-tmp\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.877865 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.877732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjd7v\" (UniqueName: \"kubernetes.io/projected/a2bd523b-e875-4aea-b1f4-def4f0875183-kube-api-access-gjd7v\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.880129 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.880104 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2bd523b-e875-4aea-b1f4-def4f0875183-tmp\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.880261 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.880225 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2bd523b-e875-4aea-b1f4-def4f0875183-tls-certs\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.890469 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.890434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjd7v\" (UniqueName: \"kubernetes.io/projected/a2bd523b-e875-4aea-b1f4-def4f0875183-kube-api-access-gjd7v\") pod \"kube-auth-proxy-6b79874b6-k8lbs\" (UID: \"a2bd523b-e875-4aea-b1f4-def4f0875183\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:09.943491 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.943412 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvl2r"] Jul 17 18:15:09.946855 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.946830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:09.949986 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.949942 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 17 18:15:09.950138 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.950111 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2vc2f\"" Jul 17 18:15:09.960201 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:09.960176 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvl2r"] Jul 17 18:15:10.011018 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.010988 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" Jul 17 18:15:10.079825 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.079660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22297a5e-a9e3-4741-9466-62f8c3f46878-cert\") pod \"odh-model-controller-7998c67dc4-hvl2r\" (UID: \"22297a5e-a9e3-4741-9466-62f8c3f46878\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.079825 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.079724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6f8\" (UniqueName: \"kubernetes.io/projected/22297a5e-a9e3-4741-9466-62f8c3f46878-kube-api-access-bn6f8\") pod \"odh-model-controller-7998c67dc4-hvl2r\" (UID: \"22297a5e-a9e3-4741-9466-62f8c3f46878\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.167336 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.167157 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs"] Jul 17 18:15:10.170503 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:15:10.170473 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bd523b_e875_4aea_b1f4_def4f0875183.slice/crio-f5c3bddf2a7a39e3d1d38359c09938693a26ce269f01443e5ec57c1d320d206a WatchSource:0}: Error finding container f5c3bddf2a7a39e3d1d38359c09938693a26ce269f01443e5ec57c1d320d206a: Status 404 returned error can't find the container with id f5c3bddf2a7a39e3d1d38359c09938693a26ce269f01443e5ec57c1d320d206a Jul 17 18:15:10.180212 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.180184 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22297a5e-a9e3-4741-9466-62f8c3f46878-cert\") pod \"odh-model-controller-7998c67dc4-hvl2r\" (UID: \"22297a5e-a9e3-4741-9466-62f8c3f46878\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.180318 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.180235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6f8\" (UniqueName: \"kubernetes.io/projected/22297a5e-a9e3-4741-9466-62f8c3f46878-kube-api-access-bn6f8\") pod \"odh-model-controller-7998c67dc4-hvl2r\" (UID: \"22297a5e-a9e3-4741-9466-62f8c3f46878\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.180386 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:15:10.180344 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 17 18:15:10.180439 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:15:10.180418 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22297a5e-a9e3-4741-9466-62f8c3f46878-cert podName:22297a5e-a9e3-4741-9466-62f8c3f46878 nodeName:}" failed. No retries permitted until 2026-07-17 18:15:10.680396898 +0000 UTC m=+321.095134913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22297a5e-a9e3-4741-9466-62f8c3f46878-cert") pod "odh-model-controller-7998c67dc4-hvl2r" (UID: "22297a5e-a9e3-4741-9466-62f8c3f46878") : secret "odh-model-controller-webhook-cert" not found Jul 17 18:15:10.190131 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.190101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6f8\" (UniqueName: \"kubernetes.io/projected/22297a5e-a9e3-4741-9466-62f8c3f46878-kube-api-access-bn6f8\") pod \"odh-model-controller-7998c67dc4-hvl2r\" (UID: \"22297a5e-a9e3-4741-9466-62f8c3f46878\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.367442 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.367351 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" event={"ID":"a2bd523b-e875-4aea-b1f4-def4f0875183","Type":"ContainerStarted","Data":"f5c3bddf2a7a39e3d1d38359c09938693a26ce269f01443e5ec57c1d320d206a"} Jul 17 18:15:10.601592 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.601557 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qbzt6"] Jul 17 18:15:10.606310 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.606279 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:10.608795 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.608773 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 17 18:15:10.624003 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.623321 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qbzt6"] Jul 17 18:15:10.683911 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.683874 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-tls-certs\") pod \"model-serving-api-54d89bd5fc-qbzt6\" (UID: \"8ab02e79-2ce3-4217-a8ac-567ae14b3f46\") " pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:10.684218 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.683918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22297a5e-a9e3-4741-9466-62f8c3f46878-cert\") pod \"odh-model-controller-7998c67dc4-hvl2r\" (UID: \"22297a5e-a9e3-4741-9466-62f8c3f46878\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.684218 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.684055 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq2wh\" (UniqueName: \"kubernetes.io/projected/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-kube-api-access-kq2wh\") pod \"model-serving-api-54d89bd5fc-qbzt6\" (UID: \"8ab02e79-2ce3-4217-a8ac-567ae14b3f46\") " pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:10.687214 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.687188 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22297a5e-a9e3-4741-9466-62f8c3f46878-cert\") pod \"odh-model-controller-7998c67dc4-hvl2r\" (UID: \"22297a5e-a9e3-4741-9466-62f8c3f46878\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.785383 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.785330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq2wh\" (UniqueName: \"kubernetes.io/projected/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-kube-api-access-kq2wh\") pod \"model-serving-api-54d89bd5fc-qbzt6\" (UID: \"8ab02e79-2ce3-4217-a8ac-567ae14b3f46\") " pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:10.785383 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.785385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-tls-certs\") pod \"model-serving-api-54d89bd5fc-qbzt6\" (UID: \"8ab02e79-2ce3-4217-a8ac-567ae14b3f46\") " pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:10.785849 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:15:10.785586 2570 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 17 18:15:10.785849 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:15:10.785665 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-tls-certs podName:8ab02e79-2ce3-4217-a8ac-567ae14b3f46 nodeName:}" failed. No retries permitted until 2026-07-17 18:15:11.285644198 +0000 UTC m=+321.700382210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-tls-certs") pod "model-serving-api-54d89bd5fc-qbzt6" (UID: "8ab02e79-2ce3-4217-a8ac-567ae14b3f46") : secret "model-serving-api-tls" not found Jul 17 18:15:10.798427 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.798388 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq2wh\" (UniqueName: \"kubernetes.io/projected/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-kube-api-access-kq2wh\") pod \"model-serving-api-54d89bd5fc-qbzt6\" (UID: \"8ab02e79-2ce3-4217-a8ac-567ae14b3f46\") " pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:10.859373 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.859337 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:10.993621 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:10.993583 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvl2r"] Jul 17 18:15:10.995525 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:15:10.995500 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22297a5e_a9e3_4741_9466_62f8c3f46878.slice/crio-d2c07dab5f6e2e7a42f89e95001e15e15870fe3659c8c835af13952d7c470003 WatchSource:0}: Error finding container d2c07dab5f6e2e7a42f89e95001e15e15870fe3659c8c835af13952d7c470003: Status 404 returned error can't find the container with id d2c07dab5f6e2e7a42f89e95001e15e15870fe3659c8c835af13952d7c470003 Jul 17 18:15:11.127350 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.127302 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5scb"] Jul 17 18:15:11.130661 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.130624 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.133316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.133206 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 18:15:11.133316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.133206 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 18:15:11.133743 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.133566 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 18:15:11.133743 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.133621 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rhvf4\"" Jul 17 18:15:11.133743 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.133679 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 18:15:11.144052 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.144016 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5scb"] Jul 17 18:15:11.289745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.289710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-token\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.289985 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.289760 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.289985 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.289794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5sm\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-kube-api-access-8z5sm\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.289985 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.289870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.289985 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.289899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.289985 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.289939 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-tls-certs\") pod \"model-serving-api-54d89bd5fc-qbzt6\" (UID: \"8ab02e79-2ce3-4217-a8ac-567ae14b3f46\") " pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:11.290336 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.289991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bdc864-7933-4d4d-8688-6610be40c7eb-local-certs\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.290336 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.290045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-cacerts\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.292818 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.292789 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab02e79-2ce3-4217-a8ac-567ae14b3f46-tls-certs\") pod \"model-serving-api-54d89bd5fc-qbzt6\" (UID: \"8ab02e79-2ce3-4217-a8ac-567ae14b3f46\") " pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:11.373385 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.373347 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" event={"ID":"22297a5e-a9e3-4741-9466-62f8c3f46878","Type":"ContainerStarted","Data":"d2c07dab5f6e2e7a42f89e95001e15e15870fe3659c8c835af13952d7c470003"} Jul 17 18:15:11.390820 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.390784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-token\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.390820 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.390827 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.391091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.390855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5sm\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-kube-api-access-8z5sm\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.391091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.390903 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.391091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.390937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.391091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.390987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bdc864-7933-4d4d-8688-6610be40c7eb-local-certs\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.391091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.391043 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-cacerts\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.391602 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.391576 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.393652 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.393609 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bdc864-7933-4d4d-8688-6610be40c7eb-local-certs\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.393840 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.393820 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.393890 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.393827 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-cacerts\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.393930 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.393879 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.398696 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.398674 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5sm\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-kube-api-access-8z5sm\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.398879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.398862 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-token\") pod \"istiod-openshift-gateway-7574459656-r5scb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.444049 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.444015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:11.526872 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.526603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:11.602133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.602098 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5scb"] Jul 17 18:15:11.605795 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:15:11.605764 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bdc864_7933_4d4d_8688_6610be40c7eb.slice/crio-e1f50de9f042f5391f7ec4f263643fe09f554053129a8860621700debd2e3f6e WatchSource:0}: Error finding container e1f50de9f042f5391f7ec4f263643fe09f554053129a8860621700debd2e3f6e: Status 404 returned error can't find the container with id e1f50de9f042f5391f7ec4f263643fe09f554053129a8860621700debd2e3f6e Jul 17 18:15:11.699066 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:11.699042 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qbzt6"] Jul 17 18:15:11.700717 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:15:11.700693 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab02e79_2ce3_4217_a8ac_567ae14b3f46.slice/crio-84b8dd3ab860cc7ee8d841d3a5e13a9f5481002b1670769624d74a3dc91814bf WatchSource:0}: Error finding container 84b8dd3ab860cc7ee8d841d3a5e13a9f5481002b1670769624d74a3dc91814bf: Status 404 returned error can't find the container with id 84b8dd3ab860cc7ee8d841d3a5e13a9f5481002b1670769624d74a3dc91814bf Jul 17 18:15:12.382280 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:12.382242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" event={"ID":"8ab02e79-2ce3-4217-a8ac-567ae14b3f46","Type":"ContainerStarted","Data":"84b8dd3ab860cc7ee8d841d3a5e13a9f5481002b1670769624d74a3dc91814bf"} Jul 17 18:15:12.384151 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:12.384095 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" event={"ID":"86bdc864-7933-4d4d-8688-6610be40c7eb","Type":"ContainerStarted","Data":"e1f50de9f042f5391f7ec4f263643fe09f554053129a8860621700debd2e3f6e"} Jul 17 18:15:16.666355 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:16.666258 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 17 18:15:16.666355 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:16.666317 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 17 18:15:17.413550 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.413519 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" event={"ID":"22297a5e-a9e3-4741-9466-62f8c3f46878","Type":"ContainerStarted","Data":"4a0f5cea16825790e9e811fd8947c16eb980a32c046facc23d934c5b32036084"} Jul 17 18:15:17.413748 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.413630 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:17.415507 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.415479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" event={"ID":"86bdc864-7933-4d4d-8688-6610be40c7eb","Type":"ContainerStarted","Data":"7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e"} Jul 17 18:15:17.415669 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.415646 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:17.417415 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.417179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" event={"ID":"a2bd523b-e875-4aea-b1f4-def4f0875183","Type":"ContainerStarted","Data":"ff1c3cb08d7570e96b6db8ed7b43b187b6ae265028ca5fd8cb01abeeaa102f9a"} Jul 17 18:15:17.417699 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.417671 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-r5scb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 18:15:17.417809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.417729 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" podUID="86bdc864-7933-4d4d-8688-6610be40c7eb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:15:17.419279 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.419256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" event={"ID":"8ab02e79-2ce3-4217-a8ac-567ae14b3f46","Type":"ContainerStarted","Data":"1703b6569f43c5552a2555f69925ec4ff417c22ca019d1919fbf9b18d1f7362d"} Jul 17 18:15:17.419399 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.419380 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:17.471036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.470969 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" podStartSLOduration=2.755123315 podStartE2EDuration="8.470939065s" totalImagesPullingTime="5.71581575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:17.468519541 +0000 UTC m=+327.883257581" watchObservedRunningTime="2026-07-17 18:15:17.470939065 +0000 UTC m=+327.885677091" Jul 17 18:15:17.511306 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.511231 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" podStartSLOduration=1.455384066 podStartE2EDuration="6.51121395s" totalImagesPullingTime="5.055829884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:17.510539095 +0000 UTC m=+327.925277159" watchObservedRunningTime="2026-07-17 18:15:17.51121395 +0000 UTC m=+327.925951977" Jul 17 18:15:17.531680 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.531614 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" podStartSLOduration=2.5726236609999997 podStartE2EDuration="7.531593173s" totalImagesPullingTime="4.958969512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:17.530973556 +0000 UTC m=+327.945711581" watchObservedRunningTime="2026-07-17 18:15:17.531593173 +0000 UTC m=+327.946331202" Jul 17 18:15:17.549085 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:17.549022 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b79874b6-k8lbs" podStartSLOduration=2.062482685 podStartE2EDuration="8.548998637s" totalImagesPullingTime="6.486515952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:17.548237907 +0000 UTC m=+327.962975934" watchObservedRunningTime="2026-07-17 18:15:17.548998637 +0000 UTC m=+327.963736667" Jul 17 18:15:18.425188 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:18.425159 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:21.408613 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.408577 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tj59h"] Jul 17 18:15:21.412049 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.412031 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.415313 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.415288 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 17 18:15:21.416018 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.416002 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mfhl5\"" Jul 17 18:15:21.434999 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.434972 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5"] Jul 17 18:15:21.438121 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.438093 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tj59h"] Jul 17 18:15:21.438271 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.438215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.441730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.441673 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 17 18:15:21.441730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.441710 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4zf7d\"" Jul 17 18:15:21.451604 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.451580 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5"] Jul 17 18:15:21.588461 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.588423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhn9z\" (UniqueName: \"kubernetes.io/projected/ac647307-f36b-449e-83ea-c4ba30e69624-kube-api-access-lhn9z\") pod \"llmisvc-controller-manager-777c6bd7d5-rdvx5\" (UID: \"ac647307-f36b-449e-83ea-c4ba30e69624\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.588461 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.588458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac647307-f36b-449e-83ea-c4ba30e69624-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rdvx5\" (UID: \"ac647307-f36b-449e-83ea-c4ba30e69624\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.588705 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.588499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f57471e8-c7c0-4122-861f-a191ed9c595c-cert\") pod \"kserve-controller-manager-d6f4b65b8-tj59h\" (UID: \"f57471e8-c7c0-4122-861f-a191ed9c595c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.588705 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.588589 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npb9\" (UniqueName: \"kubernetes.io/projected/f57471e8-c7c0-4122-861f-a191ed9c595c-kube-api-access-4npb9\") pod \"kserve-controller-manager-d6f4b65b8-tj59h\" (UID: \"f57471e8-c7c0-4122-861f-a191ed9c595c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.689305 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.689269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npb9\" (UniqueName: \"kubernetes.io/projected/f57471e8-c7c0-4122-861f-a191ed9c595c-kube-api-access-4npb9\") pod \"kserve-controller-manager-d6f4b65b8-tj59h\" (UID: \"f57471e8-c7c0-4122-861f-a191ed9c595c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.689498 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.689334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhn9z\" (UniqueName: \"kubernetes.io/projected/ac647307-f36b-449e-83ea-c4ba30e69624-kube-api-access-lhn9z\") pod \"llmisvc-controller-manager-777c6bd7d5-rdvx5\" (UID: \"ac647307-f36b-449e-83ea-c4ba30e69624\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.689498 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.689362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac647307-f36b-449e-83ea-c4ba30e69624-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rdvx5\" (UID: \"ac647307-f36b-449e-83ea-c4ba30e69624\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.689498 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.689406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f57471e8-c7c0-4122-861f-a191ed9c595c-cert\") pod \"kserve-controller-manager-d6f4b65b8-tj59h\" (UID: \"f57471e8-c7c0-4122-861f-a191ed9c595c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.691815 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.691785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f57471e8-c7c0-4122-861f-a191ed9c595c-cert\") pod \"kserve-controller-manager-d6f4b65b8-tj59h\" (UID: \"f57471e8-c7c0-4122-861f-a191ed9c595c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.691998 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.691975 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac647307-f36b-449e-83ea-c4ba30e69624-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rdvx5\" (UID: \"ac647307-f36b-449e-83ea-c4ba30e69624\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.718589 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.718558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhn9z\" (UniqueName: \"kubernetes.io/projected/ac647307-f36b-449e-83ea-c4ba30e69624-kube-api-access-lhn9z\") pod \"llmisvc-controller-manager-777c6bd7d5-rdvx5\" (UID: \"ac647307-f36b-449e-83ea-c4ba30e69624\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.718727 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.718643 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npb9\" (UniqueName: \"kubernetes.io/projected/f57471e8-c7c0-4122-861f-a191ed9c595c-kube-api-access-4npb9\") pod \"kserve-controller-manager-d6f4b65b8-tj59h\" (UID: \"f57471e8-c7c0-4122-861f-a191ed9c595c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.725368 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.725337 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:21.750653 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.750607 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:21.872085 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.872054 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tj59h"] Jul 17 18:15:21.873981 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:15:21.873934 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57471e8_c7c0_4122_861f_a191ed9c595c.slice/crio-a7212658b187af15a169e2c8d8730cc462b336efcc601b29b2b027fec0b71f79 WatchSource:0}: Error finding container a7212658b187af15a169e2c8d8730cc462b336efcc601b29b2b027fec0b71f79: Status 404 returned error can't find the container with id a7212658b187af15a169e2c8d8730cc462b336efcc601b29b2b027fec0b71f79 Jul 17 18:15:21.901639 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:21.901616 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5"] Jul 17 18:15:21.903489 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:15:21.903449 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac647307_f36b_449e_83ea_c4ba30e69624.slice/crio-36ca633b037a04ded51f7ed27bf4b909d62ad7ae850a40b92006790c987696cb WatchSource:0}: Error finding container 36ca633b037a04ded51f7ed27bf4b909d62ad7ae850a40b92006790c987696cb: Status 404 returned error can't find the container with id 36ca633b037a04ded51f7ed27bf4b909d62ad7ae850a40b92006790c987696cb Jul 17 18:15:22.444113 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:22.444069 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" event={"ID":"ac647307-f36b-449e-83ea-c4ba30e69624","Type":"ContainerStarted","Data":"36ca633b037a04ded51f7ed27bf4b909d62ad7ae850a40b92006790c987696cb"} Jul 17 18:15:22.445156 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:22.445132 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" event={"ID":"f57471e8-c7c0-4122-861f-a191ed9c595c","Type":"ContainerStarted","Data":"a7212658b187af15a169e2c8d8730cc462b336efcc601b29b2b027fec0b71f79"} Jul 17 18:15:23.805148 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:23.805112 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5scb"] Jul 17 18:15:23.805561 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:23.805360 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" podUID="86bdc864-7933-4d4d-8688-6610be40c7eb" containerName="discovery" containerID="cri-o://7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e" gracePeriod=30 Jul 17 18:15:24.053219 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.053193 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:24.114175 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.114099 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-cacerts\") pod \"86bdc864-7933-4d4d-8688-6610be40c7eb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " Jul 17 18:15:24.114175 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.114143 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-dns-cert\") pod \"86bdc864-7933-4d4d-8688-6610be40c7eb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " Jul 17 18:15:24.114175 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.114171 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bdc864-7933-4d4d-8688-6610be40c7eb-local-certs\") pod \"86bdc864-7933-4d4d-8688-6610be40c7eb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " Jul 17 18:15:24.114429 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.114202 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-kubeconfig\") pod \"86bdc864-7933-4d4d-8688-6610be40c7eb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " Jul 17 18:15:24.114429 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.114241 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-ca-configmap\") pod \"86bdc864-7933-4d4d-8688-6610be40c7eb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " Jul 17 18:15:24.114865 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.114819 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "86bdc864-7933-4d4d-8688-6610be40c7eb" (UID: "86bdc864-7933-4d4d-8688-6610be40c7eb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:15:24.116790 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.116758 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "86bdc864-7933-4d4d-8688-6610be40c7eb" (UID: "86bdc864-7933-4d4d-8688-6610be40c7eb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:15:24.116907 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.116883 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bdc864-7933-4d4d-8688-6610be40c7eb-local-certs" (OuterVolumeSpecName: "local-certs") pod "86bdc864-7933-4d4d-8688-6610be40c7eb" (UID: "86bdc864-7933-4d4d-8688-6610be40c7eb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:24.116984 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.116895 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "86bdc864-7933-4d4d-8688-6610be40c7eb" (UID: "86bdc864-7933-4d4d-8688-6610be40c7eb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:15:24.117040 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.116995 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-cacerts" (OuterVolumeSpecName: "cacerts") pod "86bdc864-7933-4d4d-8688-6610be40c7eb" (UID: "86bdc864-7933-4d4d-8688-6610be40c7eb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:15:24.214727 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.214686 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-token\") pod \"86bdc864-7933-4d4d-8688-6610be40c7eb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " Jul 17 18:15:24.214896 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.214744 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z5sm\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-kube-api-access-8z5sm\") pod \"86bdc864-7933-4d4d-8688-6610be40c7eb\" (UID: \"86bdc864-7933-4d4d-8688-6610be40c7eb\") " Jul 17 18:15:24.215016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.214971 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-cacerts\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:15:24.215016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.214992 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-dns-cert\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:15:24.215016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.215007 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bdc864-7933-4d4d-8688-6610be40c7eb-local-certs\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:15:24.215016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.215020 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-kubeconfig\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:15:24.215273 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.215033 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-csr-ca-configmap\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:15:24.217312 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.217281 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "86bdc864-7933-4d4d-8688-6610be40c7eb" (UID: "86bdc864-7933-4d4d-8688-6610be40c7eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:15:24.217523 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.217501 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-kube-api-access-8z5sm" (OuterVolumeSpecName: "kube-api-access-8z5sm") pod "86bdc864-7933-4d4d-8688-6610be40c7eb" (UID: "86bdc864-7933-4d4d-8688-6610be40c7eb"). InnerVolumeSpecName "kube-api-access-8z5sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:15:24.315450 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.315419 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-istio-token\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:15:24.315450 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.315445 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5sm\" (UniqueName: \"kubernetes.io/projected/86bdc864-7933-4d4d-8688-6610be40c7eb-kube-api-access-8z5sm\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:15:24.455415 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.455371 2570 generic.go:356] "Generic (PLEG): container finished" podID="86bdc864-7933-4d4d-8688-6610be40c7eb" containerID="7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e" exitCode=0 Jul 17 18:15:24.455590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.455425 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" Jul 17 18:15:24.455590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.455425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" event={"ID":"86bdc864-7933-4d4d-8688-6610be40c7eb","Type":"ContainerDied","Data":"7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e"} Jul 17 18:15:24.455590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.455532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5scb" event={"ID":"86bdc864-7933-4d4d-8688-6610be40c7eb","Type":"ContainerDied","Data":"e1f50de9f042f5391f7ec4f263643fe09f554053129a8860621700debd2e3f6e"} Jul 17 18:15:24.455590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.455559 2570 scope.go:117] "RemoveContainer" containerID="7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e" Jul 17 18:15:24.464854 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.464833 2570 scope.go:117] "RemoveContainer" containerID="7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e" Jul 17 18:15:24.465273 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:15:24.465242 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e\": container with ID starting with 7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e not found: ID does not exist" containerID="7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e" Jul 17 18:15:24.465354 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.465283 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e"} err="failed to get container status \"7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e\": rpc error: code = NotFound desc = could not find container \"7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e\": container with ID starting with 7ad55343fea554b12e2f9f503dad4ff467de73b0735d69df36254fad141c9a6e not found: ID does not exist" Jul 17 18:15:24.479052 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.479024 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5scb"] Jul 17 18:15:24.483363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:24.483341 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5scb"] Jul 17 18:15:26.071904 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:26.071822 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86bdc864-7933-4d4d-8688-6610be40c7eb" path="/var/lib/kubelet/pods/86bdc864-7933-4d4d-8688-6610be40c7eb/volumes" Jul 17 18:15:26.468083 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:26.468049 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" event={"ID":"f57471e8-c7c0-4122-861f-a191ed9c595c","Type":"ContainerStarted","Data":"fcd17c0cf62b1f4c5b0f6b785f42278b9e469d95f732f0e5e0c3bc4b640bd8b6"} Jul 17 18:15:26.468303 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:26.468166 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:15:26.469462 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:26.469431 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" event={"ID":"ac647307-f36b-449e-83ea-c4ba30e69624","Type":"ContainerStarted","Data":"b309b35c50d759ea7446ed956701c1b5fbb14e3f55347a7273485ea6080da6f7"} Jul 17 18:15:26.469573 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:26.469564 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:26.486440 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:26.486392 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" podStartSLOduration=1.5863906920000002 podStartE2EDuration="5.486377835s" totalImagesPullingTime="3.899987143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:26.484711838 +0000 UTC m=+336.899449865" watchObservedRunningTime="2026-07-17 18:15:26.486377835 +0000 UTC m=+336.901115862" Jul 17 18:15:26.501326 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:26.501274 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" podStartSLOduration=1.627198612 podStartE2EDuration="5.501257836s" totalImagesPullingTime="3.874059224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:26.499498639 +0000 UTC m=+336.914236667" watchObservedRunningTime="2026-07-17 18:15:26.501257836 +0000 UTC m=+336.915995867" Jul 17 18:15:28.426447 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:28.426420 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hvl2r" Jul 17 18:15:28.429970 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:28.429929 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qbzt6" Jul 17 18:15:57.480758 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:57.480728 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rdvx5" Jul 17 18:15:57.483937 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:15:57.483914 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tj59h" Jul 17 18:16:10.844914 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.844881 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ct2fz"] Jul 17 18:16:10.845332 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.845164 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86bdc864-7933-4d4d-8688-6610be40c7eb" containerName="discovery" Jul 17 18:16:10.845332 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.845174 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="86bdc864-7933-4d4d-8688-6610be40c7eb" containerName="discovery" Jul 17 18:16:10.845332 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.845233 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="86bdc864-7933-4d4d-8688-6610be40c7eb" containerName="discovery" Jul 17 18:16:10.854674 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.854642 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" Jul 17 18:16:10.857567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.857535 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ct2fz"] Jul 17 18:16:10.857727 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.857680 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 18:16:10.858892 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.858862 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xl4xb\"" Jul 17 18:16:10.858892 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.858871 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 18:16:10.983643 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:10.983608 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6pf\" (UniqueName: \"kubernetes.io/projected/729eb0f3-738c-42f0-a916-5a9aa57355a0-kube-api-access-mn6pf\") pod \"kuadrant-operator-catalog-ct2fz\" (UID: \"729eb0f3-738c-42f0-a916-5a9aa57355a0\") " pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" Jul 17 18:16:11.084840 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.084793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6pf\" (UniqueName: \"kubernetes.io/projected/729eb0f3-738c-42f0-a916-5a9aa57355a0-kube-api-access-mn6pf\") pod \"kuadrant-operator-catalog-ct2fz\" (UID: \"729eb0f3-738c-42f0-a916-5a9aa57355a0\") " pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" Jul 17 18:16:11.093578 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.093549 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6pf\" (UniqueName: \"kubernetes.io/projected/729eb0f3-738c-42f0-a916-5a9aa57355a0-kube-api-access-mn6pf\") pod \"kuadrant-operator-catalog-ct2fz\" (UID: \"729eb0f3-738c-42f0-a916-5a9aa57355a0\") " pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" Jul 17 18:16:11.165646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.165552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" Jul 17 18:16:11.211836 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.211808 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ct2fz"] Jul 17 18:16:11.296458 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.296434 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ct2fz"] Jul 17 18:16:11.298215 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:11.298188 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729eb0f3_738c_42f0_a916_5a9aa57355a0.slice/crio-1ce359577281011cc0b91986778fa1df5049c08e59dda00b49be91f6db369db6 WatchSource:0}: Error finding container 1ce359577281011cc0b91986778fa1df5049c08e59dda00b49be91f6db369db6: Status 404 returned error can't find the container with id 1ce359577281011cc0b91986778fa1df5049c08e59dda00b49be91f6db369db6 Jul 17 18:16:11.427787 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.427740 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lc78v"] Jul 17 18:16:11.432205 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.432182 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:11.440353 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.440329 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lc78v"] Jul 17 18:16:11.589441 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.589404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbj6\" (UniqueName: \"kubernetes.io/projected/cf5e7b80-d007-4d25-a6ea-f59634b988cd-kube-api-access-tsbj6\") pod \"kuadrant-operator-catalog-lc78v\" (UID: \"cf5e7b80-d007-4d25-a6ea-f59634b988cd\") " pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:11.664300 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.664264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" event={"ID":"729eb0f3-738c-42f0-a916-5a9aa57355a0","Type":"ContainerStarted","Data":"1ce359577281011cc0b91986778fa1df5049c08e59dda00b49be91f6db369db6"} Jul 17 18:16:11.690687 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.690604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbj6\" (UniqueName: \"kubernetes.io/projected/cf5e7b80-d007-4d25-a6ea-f59634b988cd-kube-api-access-tsbj6\") pod \"kuadrant-operator-catalog-lc78v\" (UID: \"cf5e7b80-d007-4d25-a6ea-f59634b988cd\") " pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:11.699402 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.699364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbj6\" (UniqueName: \"kubernetes.io/projected/cf5e7b80-d007-4d25-a6ea-f59634b988cd-kube-api-access-tsbj6\") pod \"kuadrant-operator-catalog-lc78v\" (UID: \"cf5e7b80-d007-4d25-a6ea-f59634b988cd\") " pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:11.743021 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.742988 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:11.874382 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:11.874200 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lc78v"] Jul 17 18:16:11.956148 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:11.956074 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5e7b80_d007_4d25_a6ea_f59634b988cd.slice/crio-0ebfbc01ca753488cfe56964508c3550bed6867b21383aa09b50e5dae33c4739 WatchSource:0}: Error finding container 0ebfbc01ca753488cfe56964508c3550bed6867b21383aa09b50e5dae33c4739: Status 404 returned error can't find the container with id 0ebfbc01ca753488cfe56964508c3550bed6867b21383aa09b50e5dae33c4739 Jul 17 18:16:12.676183 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:12.676150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" event={"ID":"cf5e7b80-d007-4d25-a6ea-f59634b988cd","Type":"ContainerStarted","Data":"0ebfbc01ca753488cfe56964508c3550bed6867b21383aa09b50e5dae33c4739"} Jul 17 18:16:13.682694 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:13.682657 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" event={"ID":"cf5e7b80-d007-4d25-a6ea-f59634b988cd","Type":"ContainerStarted","Data":"f3e7c00c179dd9d2565a22c8e577e0bacc8e645373980b4e62977d1292eeb11a"} Jul 17 18:16:13.684018 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:13.683992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" event={"ID":"729eb0f3-738c-42f0-a916-5a9aa57355a0","Type":"ContainerStarted","Data":"256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7"} Jul 17 18:16:13.684133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:13.684061 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" podUID="729eb0f3-738c-42f0-a916-5a9aa57355a0" containerName="registry-server" containerID="cri-o://256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7" gracePeriod=2 Jul 17 18:16:13.701547 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:13.701504 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" podStartSLOduration=1.495767108 podStartE2EDuration="2.701488235s" totalImagesPullingTime="1.205721127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:13.6991366 +0000 UTC m=+384.113874628" watchObservedRunningTime="2026-07-17 18:16:13.701488235 +0000 UTC m=+384.116226261" Jul 17 18:16:13.715618 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:13.715573 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" podStartSLOduration=1.8525251329999999 podStartE2EDuration="3.715555557s" totalImagesPullingTime="1.863030424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:13.714764188 +0000 UTC m=+384.129502218" watchObservedRunningTime="2026-07-17 18:16:13.715555557 +0000 UTC m=+384.130293584" Jul 17 18:16:13.927822 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:13.927798 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" Jul 17 18:16:14.010950 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.010862 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn6pf\" (UniqueName: \"kubernetes.io/projected/729eb0f3-738c-42f0-a916-5a9aa57355a0-kube-api-access-mn6pf\") pod \"729eb0f3-738c-42f0-a916-5a9aa57355a0\" (UID: \"729eb0f3-738c-42f0-a916-5a9aa57355a0\") " Jul 17 18:16:14.013096 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.013069 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729eb0f3-738c-42f0-a916-5a9aa57355a0-kube-api-access-mn6pf" (OuterVolumeSpecName: "kube-api-access-mn6pf") pod "729eb0f3-738c-42f0-a916-5a9aa57355a0" (UID: "729eb0f3-738c-42f0-a916-5a9aa57355a0"). InnerVolumeSpecName "kube-api-access-mn6pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:16:14.111800 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.111766 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6pf\" (UniqueName: \"kubernetes.io/projected/729eb0f3-738c-42f0-a916-5a9aa57355a0-kube-api-access-mn6pf\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:14.689791 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.689759 2570 generic.go:356] "Generic (PLEG): container finished" podID="729eb0f3-738c-42f0-a916-5a9aa57355a0" containerID="256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7" exitCode=0 Jul 17 18:16:14.690265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.689806 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" Jul 17 18:16:14.690265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.689848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" event={"ID":"729eb0f3-738c-42f0-a916-5a9aa57355a0","Type":"ContainerDied","Data":"256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7"} Jul 17 18:16:14.690265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.689898 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ct2fz" event={"ID":"729eb0f3-738c-42f0-a916-5a9aa57355a0","Type":"ContainerDied","Data":"1ce359577281011cc0b91986778fa1df5049c08e59dda00b49be91f6db369db6"} Jul 17 18:16:14.690265 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.689924 2570 scope.go:117] "RemoveContainer" containerID="256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7" Jul 17 18:16:14.698678 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.698654 2570 scope.go:117] "RemoveContainer" containerID="256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7" Jul 17 18:16:14.699026 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:16:14.699007 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7\": container with ID starting with 256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7 not found: ID does not exist" containerID="256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7" Jul 17 18:16:14.699097 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.699036 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7"} err="failed to get container status \"256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7\": rpc error: code = NotFound desc = could not find container \"256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7\": container with ID starting with 256615ef9d66dde06fb1bb37eb5b973700b1e2a4213fe42566c17b91ed16c4e7 not found: ID does not exist" Jul 17 18:16:14.707020 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.706991 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ct2fz"] Jul 17 18:16:14.711435 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:14.711410 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ct2fz"] Jul 17 18:16:16.072062 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:16.072022 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729eb0f3-738c-42f0-a916-5a9aa57355a0" path="/var/lib/kubelet/pods/729eb0f3-738c-42f0-a916-5a9aa57355a0/volumes" Jul 17 18:16:21.743874 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:21.743834 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:21.744443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:21.743889 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:21.767014 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:21.766988 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:22.747381 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:22.747353 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-lc78v" Jul 17 18:16:26.266941 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.266908 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m"] Jul 17 18:16:26.267336 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.267209 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729eb0f3-738c-42f0-a916-5a9aa57355a0" containerName="registry-server" Jul 17 18:16:26.267336 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.267221 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="729eb0f3-738c-42f0-a916-5a9aa57355a0" containerName="registry-server" Jul 17 18:16:26.267336 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.267265 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="729eb0f3-738c-42f0-a916-5a9aa57355a0" containerName="registry-server" Jul 17 18:16:26.271899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.271881 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.274747 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.274726 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zgbg7\"" Jul 17 18:16:26.284168 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.284143 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m"] Jul 17 18:16:26.412029 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.411994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.412219 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.412056 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.412219 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.412126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpmm\" (UniqueName: \"kubernetes.io/projected/36c88212-fb53-414b-87a7-a412c0aab5a1-kube-api-access-vkpmm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.512560 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.512522 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.512748 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.512592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.512748 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.512634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpmm\" (UniqueName: \"kubernetes.io/projected/36c88212-fb53-414b-87a7-a412c0aab5a1-kube-api-access-vkpmm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.512913 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.512893 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.513005 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.512930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.523180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.523113 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpmm\" (UniqueName: \"kubernetes.io/projected/36c88212-fb53-414b-87a7-a412c0aab5a1-kube-api-access-vkpmm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.580695 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.580657 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:26.712021 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.711996 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m"] Jul 17 18:16:26.714171 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:26.714144 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c88212_fb53_414b_87a7_a412c0aab5a1.slice/crio-093f6165a0670affcd5d2a586bd6b463ff0c74b1dd7db7ec6ff6f6321e77710d WatchSource:0}: Error finding container 093f6165a0670affcd5d2a586bd6b463ff0c74b1dd7db7ec6ff6f6321e77710d: Status 404 returned error can't find the container with id 093f6165a0670affcd5d2a586bd6b463ff0c74b1dd7db7ec6ff6f6321e77710d Jul 17 18:16:26.743221 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.743186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" event={"ID":"36c88212-fb53-414b-87a7-a412c0aab5a1","Type":"ContainerStarted","Data":"093f6165a0670affcd5d2a586bd6b463ff0c74b1dd7db7ec6ff6f6321e77710d"} Jul 17 18:16:26.896720 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.896684 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs"] Jul 17 18:16:26.900134 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.900116 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:26.931879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:26.931852 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs"] Jul 17 18:16:27.016261 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.016229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvks\" (UniqueName: \"kubernetes.io/projected/938899aa-38b1-42d7-a873-df2b0aef42eb-kube-api-access-nxvks\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.016431 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.016268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.016431 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.016387 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.117647 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.117539 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.117647 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.117636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.117647 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.117658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvks\" (UniqueName: \"kubernetes.io/projected/938899aa-38b1-42d7-a873-df2b0aef42eb-kube-api-access-nxvks\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.117981 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.117933 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.118078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.118062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.126872 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.126853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvks\" (UniqueName: \"kubernetes.io/projected/938899aa-38b1-42d7-a873-df2b0aef42eb-kube-api-access-nxvks\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.208790 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.208754 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:27.253454 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.253424 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz"] Jul 17 18:16:27.258340 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.258032 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.270837 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.270812 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz"] Jul 17 18:16:27.352561 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.352540 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs"] Jul 17 18:16:27.354830 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:27.354800 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938899aa_38b1_42d7_a873_df2b0aef42eb.slice/crio-fb45775145f39d3b707c0d940f160b95c75f506a849bfe21d520f62b03f7add1 WatchSource:0}: Error finding container fb45775145f39d3b707c0d940f160b95c75f506a849bfe21d520f62b03f7add1: Status 404 returned error can't find the container with id fb45775145f39d3b707c0d940f160b95c75f506a849bfe21d520f62b03f7add1 Jul 17 18:16:27.419802 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.419775 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.419914 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.419826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrw9\" (UniqueName: \"kubernetes.io/projected/273b7e4d-641c-4591-a119-cc72059acf9d-kube-api-access-zhrw9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.419914 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.419860 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.520815 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.520777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.521037 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.520839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrw9\" (UniqueName: \"kubernetes.io/projected/273b7e4d-641c-4591-a119-cc72059acf9d-kube-api-access-zhrw9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.521037 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.520871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.521200 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.521177 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.521267 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.521229 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.535652 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.535627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrw9\" (UniqueName: \"kubernetes.io/projected/273b7e4d-641c-4591-a119-cc72059acf9d-kube-api-access-zhrw9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.571640 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.571607 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:27.716866 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.716829 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n"] Jul 17 18:16:27.721252 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.721235 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.735658 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.735628 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n"] Jul 17 18:16:27.739465 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.739431 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz"] Jul 17 18:16:27.741510 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:27.741485 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273b7e4d_641c_4591_a119_cc72059acf9d.slice/crio-5566477e9810b8a0b74012fe05e560aa0ddf555efcd30ac31c3d68ed33950c2b WatchSource:0}: Error finding container 5566477e9810b8a0b74012fe05e560aa0ddf555efcd30ac31c3d68ed33950c2b: Status 404 returned error can't find the container with id 5566477e9810b8a0b74012fe05e560aa0ddf555efcd30ac31c3d68ed33950c2b Jul 17 18:16:27.748907 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.748879 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" event={"ID":"273b7e4d-641c-4591-a119-cc72059acf9d","Type":"ContainerStarted","Data":"5566477e9810b8a0b74012fe05e560aa0ddf555efcd30ac31c3d68ed33950c2b"} Jul 17 18:16:27.750546 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.750521 2570 generic.go:356] "Generic (PLEG): container finished" podID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerID="9d867f7ba4414957f58c7a0d9050a4a75f488a579343fefa05b449f1d629f863" exitCode=0 Jul 17 18:16:27.750686 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.750610 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" event={"ID":"938899aa-38b1-42d7-a873-df2b0aef42eb","Type":"ContainerDied","Data":"9d867f7ba4414957f58c7a0d9050a4a75f488a579343fefa05b449f1d629f863"} Jul 17 18:16:27.750686 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.750657 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" event={"ID":"938899aa-38b1-42d7-a873-df2b0aef42eb","Type":"ContainerStarted","Data":"fb45775145f39d3b707c0d940f160b95c75f506a849bfe21d520f62b03f7add1"} Jul 17 18:16:27.752309 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.752275 2570 generic.go:356] "Generic (PLEG): container finished" podID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerID="78254e09d5091daac45f09d8795cb3e9509feeb95fbbe49ce344a06bb6b1722c" exitCode=0 Jul 17 18:16:27.752408 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.752324 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" event={"ID":"36c88212-fb53-414b-87a7-a412c0aab5a1","Type":"ContainerDied","Data":"78254e09d5091daac45f09d8795cb3e9509feeb95fbbe49ce344a06bb6b1722c"} Jul 17 18:16:27.823420 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.823379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.823588 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.823463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.823588 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.823545 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-778jt\" (UniqueName: \"kubernetes.io/projected/a6441300-acfd-4e7e-87bc-463e6baccaa3-kube-api-access-778jt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.924426 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.924329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.924604 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.924445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-778jt\" (UniqueName: \"kubernetes.io/projected/a6441300-acfd-4e7e-87bc-463e6baccaa3-kube-api-access-778jt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.924604 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.924560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.924826 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.924809 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.924889 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.924867 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:27.933824 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:27.933803 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-778jt\" (UniqueName: \"kubernetes.io/projected/a6441300-acfd-4e7e-87bc-463e6baccaa3-kube-api-access-778jt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:28.029908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.029870 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:28.162709 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.162683 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n"] Jul 17 18:16:28.164155 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:28.164132 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6441300_acfd_4e7e_87bc_463e6baccaa3.slice/crio-442f2fe6757e800935598301b37fe609c384631412ec73f824c3cd51bd03bc4a WatchSource:0}: Error finding container 442f2fe6757e800935598301b37fe609c384631412ec73f824c3cd51bd03bc4a: Status 404 returned error can't find the container with id 442f2fe6757e800935598301b37fe609c384631412ec73f824c3cd51bd03bc4a Jul 17 18:16:28.758343 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.758315 2570 generic.go:356] "Generic (PLEG): container finished" podID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerID="e60030e3a68c0a8ce21c7898fed63fda5cceeb2d676a2f6a66c518e1aa70b1b8" exitCode=0 Jul 17 18:16:28.758743 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.758399 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" event={"ID":"a6441300-acfd-4e7e-87bc-463e6baccaa3","Type":"ContainerDied","Data":"e60030e3a68c0a8ce21c7898fed63fda5cceeb2d676a2f6a66c518e1aa70b1b8"} Jul 17 18:16:28.758743 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.758451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" event={"ID":"a6441300-acfd-4e7e-87bc-463e6baccaa3","Type":"ContainerStarted","Data":"442f2fe6757e800935598301b37fe609c384631412ec73f824c3cd51bd03bc4a"} Jul 17 18:16:28.760073 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.760041 2570 generic.go:356] "Generic (PLEG): container finished" podID="273b7e4d-641c-4591-a119-cc72059acf9d" containerID="0c364935c2cb3737bf5e21e8f880f7491f978b7b420ca9e9e018d77c25ac71fe" exitCode=0 Jul 17 18:16:28.760201 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.760128 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" event={"ID":"273b7e4d-641c-4591-a119-cc72059acf9d","Type":"ContainerDied","Data":"0c364935c2cb3737bf5e21e8f880f7491f978b7b420ca9e9e018d77c25ac71fe"} Jul 17 18:16:28.761923 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.761905 2570 generic.go:356] "Generic (PLEG): container finished" podID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerID="0be5195679934d1cee08269c917a3ed3696b8938f00e8a98ad7d29ff00d59b02" exitCode=0 Jul 17 18:16:28.762030 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.761995 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" event={"ID":"938899aa-38b1-42d7-a873-df2b0aef42eb","Type":"ContainerDied","Data":"0be5195679934d1cee08269c917a3ed3696b8938f00e8a98ad7d29ff00d59b02"} Jul 17 18:16:28.764017 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.763993 2570 generic.go:356] "Generic (PLEG): container finished" podID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerID="afe8fbf63b2753a01f33467b8a7e428b23683517f07b3ff4c8264a40832c457c" exitCode=0 Jul 17 18:16:28.764135 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:28.764027 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" event={"ID":"36c88212-fb53-414b-87a7-a412c0aab5a1","Type":"ContainerDied","Data":"afe8fbf63b2753a01f33467b8a7e428b23683517f07b3ff4c8264a40832c457c"} Jul 17 18:16:29.771220 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:29.771193 2570 generic.go:356] "Generic (PLEG): container finished" podID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerID="a71f1317296c0e656df99accc4abe4ed22cb792c49f04f966475a8bd797dede3" exitCode=0 Jul 17 18:16:29.771623 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:29.771277 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" event={"ID":"938899aa-38b1-42d7-a873-df2b0aef42eb","Type":"ContainerDied","Data":"a71f1317296c0e656df99accc4abe4ed22cb792c49f04f966475a8bd797dede3"} Jul 17 18:16:29.773182 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:29.773161 2570 generic.go:356] "Generic (PLEG): container finished" podID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerID="d3048e0470e72e4bcc0ade5bb858140eb3b9a27b72ec0290f7987b53d8a2e371" exitCode=0 Jul 17 18:16:29.773277 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:29.773235 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" event={"ID":"36c88212-fb53-414b-87a7-a412c0aab5a1","Type":"ContainerDied","Data":"d3048e0470e72e4bcc0ade5bb858140eb3b9a27b72ec0290f7987b53d8a2e371"} Jul 17 18:16:30.779827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:30.779788 2570 generic.go:356] "Generic (PLEG): container finished" podID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerID="eb87345da6dc9f28382de11d0a612034c1b3f036e8347cd7f2d073af84d9b735" exitCode=0 Jul 17 18:16:30.780296 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:30.779877 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" event={"ID":"a6441300-acfd-4e7e-87bc-463e6baccaa3","Type":"ContainerDied","Data":"eb87345da6dc9f28382de11d0a612034c1b3f036e8347cd7f2d073af84d9b735"} Jul 17 18:16:30.781696 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:30.781676 2570 generic.go:356] "Generic (PLEG): container finished" podID="273b7e4d-641c-4591-a119-cc72059acf9d" containerID="76c5306085642637fc8d9ac468d59278329717293d6133c142ef081f6378878f" exitCode=0 Jul 17 18:16:30.781780 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:30.781704 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" event={"ID":"273b7e4d-641c-4591-a119-cc72059acf9d","Type":"ContainerDied","Data":"76c5306085642637fc8d9ac468d59278329717293d6133c142ef081f6378878f"} Jul 17 18:16:30.931297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:30.931271 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:30.961398 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:30.961335 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:31.050488 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.050402 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-util\") pod \"938899aa-38b1-42d7-a873-df2b0aef42eb\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " Jul 17 18:16:31.050488 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.050445 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-bundle\") pod \"36c88212-fb53-414b-87a7-a412c0aab5a1\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " Jul 17 18:16:31.050488 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.050473 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-bundle\") pod \"938899aa-38b1-42d7-a873-df2b0aef42eb\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " Jul 17 18:16:31.050731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.050536 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxvks\" (UniqueName: \"kubernetes.io/projected/938899aa-38b1-42d7-a873-df2b0aef42eb-kube-api-access-nxvks\") pod \"938899aa-38b1-42d7-a873-df2b0aef42eb\" (UID: \"938899aa-38b1-42d7-a873-df2b0aef42eb\") " Jul 17 18:16:31.050731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.050563 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkpmm\" (UniqueName: \"kubernetes.io/projected/36c88212-fb53-414b-87a7-a412c0aab5a1-kube-api-access-vkpmm\") pod \"36c88212-fb53-414b-87a7-a412c0aab5a1\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " Jul 17 18:16:31.050731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.050606 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-util\") pod \"36c88212-fb53-414b-87a7-a412c0aab5a1\" (UID: \"36c88212-fb53-414b-87a7-a412c0aab5a1\") " Jul 17 18:16:31.051034 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.051003 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-bundle" (OuterVolumeSpecName: "bundle") pod "36c88212-fb53-414b-87a7-a412c0aab5a1" (UID: "36c88212-fb53-414b-87a7-a412c0aab5a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:31.051109 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.051040 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-bundle" (OuterVolumeSpecName: "bundle") pod "938899aa-38b1-42d7-a873-df2b0aef42eb" (UID: "938899aa-38b1-42d7-a873-df2b0aef42eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:31.052765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.052732 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c88212-fb53-414b-87a7-a412c0aab5a1-kube-api-access-vkpmm" (OuterVolumeSpecName: "kube-api-access-vkpmm") pod "36c88212-fb53-414b-87a7-a412c0aab5a1" (UID: "36c88212-fb53-414b-87a7-a412c0aab5a1"). InnerVolumeSpecName "kube-api-access-vkpmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:16:31.052765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.052750 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938899aa-38b1-42d7-a873-df2b0aef42eb-kube-api-access-nxvks" (OuterVolumeSpecName: "kube-api-access-nxvks") pod "938899aa-38b1-42d7-a873-df2b0aef42eb" (UID: "938899aa-38b1-42d7-a873-df2b0aef42eb"). InnerVolumeSpecName "kube-api-access-nxvks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:16:31.059817 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.059793 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-util" (OuterVolumeSpecName: "util") pod "938899aa-38b1-42d7-a873-df2b0aef42eb" (UID: "938899aa-38b1-42d7-a873-df2b0aef42eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:31.060076 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.060060 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-util" (OuterVolumeSpecName: "util") pod "36c88212-fb53-414b-87a7-a412c0aab5a1" (UID: "36c88212-fb53-414b-87a7-a412c0aab5a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:31.151730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.151693 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxvks\" (UniqueName: \"kubernetes.io/projected/938899aa-38b1-42d7-a873-df2b0aef42eb-kube-api-access-nxvks\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:31.151730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.151722 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpmm\" (UniqueName: \"kubernetes.io/projected/36c88212-fb53-414b-87a7-a412c0aab5a1-kube-api-access-vkpmm\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:31.151730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.151732 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-util\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:31.151730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.151741 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-util\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:31.152019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.151749 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c88212-fb53-414b-87a7-a412c0aab5a1-bundle\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:31.152019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.151756 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938899aa-38b1-42d7-a873-df2b0aef42eb-bundle\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:31.788362 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.788333 2570 generic.go:356] "Generic (PLEG): container finished" podID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerID="7dd8046b9bf7d64f4740584de184ee323c7a278ff65b2a56594720cb96e273d8" exitCode=0 Jul 17 18:16:31.788788 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.788421 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" event={"ID":"a6441300-acfd-4e7e-87bc-463e6baccaa3","Type":"ContainerDied","Data":"7dd8046b9bf7d64f4740584de184ee323c7a278ff65b2a56594720cb96e273d8"} Jul 17 18:16:31.790387 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.790363 2570 generic.go:356] "Generic (PLEG): container finished" podID="273b7e4d-641c-4591-a119-cc72059acf9d" containerID="6004ea0cd163f4268667b6b719ae504584e9175db8fcba89befbaa08509a42fa" exitCode=0 Jul 17 18:16:31.790513 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.790425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" event={"ID":"273b7e4d-641c-4591-a119-cc72059acf9d","Type":"ContainerDied","Data":"6004ea0cd163f4268667b6b719ae504584e9175db8fcba89befbaa08509a42fa"} Jul 17 18:16:31.792253 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.792236 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" Jul 17 18:16:31.792346 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.792258 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs" event={"ID":"938899aa-38b1-42d7-a873-df2b0aef42eb","Type":"ContainerDied","Data":"fb45775145f39d3b707c0d940f160b95c75f506a849bfe21d520f62b03f7add1"} Jul 17 18:16:31.792346 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.792294 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb45775145f39d3b707c0d940f160b95c75f506a849bfe21d520f62b03f7add1" Jul 17 18:16:31.794163 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.794143 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" event={"ID":"36c88212-fb53-414b-87a7-a412c0aab5a1","Type":"ContainerDied","Data":"093f6165a0670affcd5d2a586bd6b463ff0c74b1dd7db7ec6ff6f6321e77710d"} Jul 17 18:16:31.794252 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.794175 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="093f6165a0670affcd5d2a586bd6b463ff0c74b1dd7db7ec6ff6f6321e77710d" Jul 17 18:16:31.794252 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:31.794176 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m" Jul 17 18:16:32.941893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:32.941868 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:32.974054 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:32.974031 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:33.066459 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.066368 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhrw9\" (UniqueName: \"kubernetes.io/projected/273b7e4d-641c-4591-a119-cc72059acf9d-kube-api-access-zhrw9\") pod \"273b7e4d-641c-4591-a119-cc72059acf9d\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " Jul 17 18:16:33.066459 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.066418 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-util\") pod \"273b7e4d-641c-4591-a119-cc72059acf9d\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " Jul 17 18:16:33.066687 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.066492 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-bundle\") pod \"273b7e4d-641c-4591-a119-cc72059acf9d\" (UID: \"273b7e4d-641c-4591-a119-cc72059acf9d\") " Jul 17 18:16:33.067031 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.067009 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-bundle" (OuterVolumeSpecName: "bundle") pod "273b7e4d-641c-4591-a119-cc72059acf9d" (UID: "273b7e4d-641c-4591-a119-cc72059acf9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:33.068426 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.068400 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273b7e4d-641c-4591-a119-cc72059acf9d-kube-api-access-zhrw9" (OuterVolumeSpecName: "kube-api-access-zhrw9") pod "273b7e4d-641c-4591-a119-cc72059acf9d" (UID: "273b7e4d-641c-4591-a119-cc72059acf9d"). InnerVolumeSpecName "kube-api-access-zhrw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:16:33.073943 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.073910 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-util" (OuterVolumeSpecName: "util") pod "273b7e4d-641c-4591-a119-cc72059acf9d" (UID: "273b7e4d-641c-4591-a119-cc72059acf9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:33.167928 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.167876 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-778jt\" (UniqueName: \"kubernetes.io/projected/a6441300-acfd-4e7e-87bc-463e6baccaa3-kube-api-access-778jt\") pod \"a6441300-acfd-4e7e-87bc-463e6baccaa3\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " Jul 17 18:16:33.167928 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.167936 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-bundle\") pod \"a6441300-acfd-4e7e-87bc-463e6baccaa3\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " Jul 17 18:16:33.168188 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.167994 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-util\") pod \"a6441300-acfd-4e7e-87bc-463e6baccaa3\" (UID: \"a6441300-acfd-4e7e-87bc-463e6baccaa3\") " Jul 17 18:16:33.168188 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.168175 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-bundle\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:33.168188 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.168186 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrw9\" (UniqueName: \"kubernetes.io/projected/273b7e4d-641c-4591-a119-cc72059acf9d-kube-api-access-zhrw9\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:33.168362 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.168196 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/273b7e4d-641c-4591-a119-cc72059acf9d-util\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:33.168544 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.168516 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-bundle" (OuterVolumeSpecName: "bundle") pod "a6441300-acfd-4e7e-87bc-463e6baccaa3" (UID: "a6441300-acfd-4e7e-87bc-463e6baccaa3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:33.169950 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.169927 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6441300-acfd-4e7e-87bc-463e6baccaa3-kube-api-access-778jt" (OuterVolumeSpecName: "kube-api-access-778jt") pod "a6441300-acfd-4e7e-87bc-463e6baccaa3" (UID: "a6441300-acfd-4e7e-87bc-463e6baccaa3"). InnerVolumeSpecName "kube-api-access-778jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:16:33.173626 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.173598 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-util" (OuterVolumeSpecName: "util") pod "a6441300-acfd-4e7e-87bc-463e6baccaa3" (UID: "a6441300-acfd-4e7e-87bc-463e6baccaa3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:16:33.268714 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.268674 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-778jt\" (UniqueName: \"kubernetes.io/projected/a6441300-acfd-4e7e-87bc-463e6baccaa3-kube-api-access-778jt\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:33.268714 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.268709 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-bundle\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:33.268714 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.268721 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6441300-acfd-4e7e-87bc-463e6baccaa3-util\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:16:33.807242 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.807217 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" Jul 17 18:16:33.807242 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.807217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n" event={"ID":"a6441300-acfd-4e7e-87bc-463e6baccaa3","Type":"ContainerDied","Data":"442f2fe6757e800935598301b37fe609c384631412ec73f824c3cd51bd03bc4a"} Jul 17 18:16:33.807461 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.807267 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442f2fe6757e800935598301b37fe609c384631412ec73f824c3cd51bd03bc4a" Jul 17 18:16:33.809283 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.809257 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" event={"ID":"273b7e4d-641c-4591-a119-cc72059acf9d","Type":"ContainerDied","Data":"5566477e9810b8a0b74012fe05e560aa0ddf555efcd30ac31c3d68ed33950c2b"} Jul 17 18:16:33.809402 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.809295 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5566477e9810b8a0b74012fe05e560aa0ddf555efcd30ac31c3d68ed33950c2b" Jul 17 18:16:33.809402 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:33.809269 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz" Jul 17 18:16:44.675395 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675364 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l"] Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675667 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerName="util" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675678 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerName="util" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675692 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerName="util" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675698 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerName="util" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675707 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerName="extract" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675714 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerName="extract" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675722 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675727 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675732 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273b7e4d-641c-4591-a119-cc72059acf9d" containerName="extract" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675737 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="273b7e4d-641c-4591-a119-cc72059acf9d" containerName="extract" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675743 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675748 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675755 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerName="extract" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675760 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerName="extract" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675765 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273b7e4d-641c-4591-a119-cc72059acf9d" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675770 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="273b7e4d-641c-4591-a119-cc72059acf9d" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675781 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerName="util" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675786 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerName="util" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675791 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675796 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerName="pull" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675801 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273b7e4d-641c-4591-a119-cc72059acf9d" containerName="util" Jul 17 18:16:44.675794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675805 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="273b7e4d-641c-4591-a119-cc72059acf9d" containerName="util" Jul 17 18:16:44.676414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675810 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerName="extract" Jul 17 18:16:44.676414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675815 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerName="extract" Jul 17 18:16:44.676414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675859 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c88212-fb53-414b-87a7-a412c0aab5a1" containerName="extract" Jul 17 18:16:44.676414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675867 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="273b7e4d-641c-4591-a119-cc72059acf9d" containerName="extract" Jul 17 18:16:44.676414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675871 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="938899aa-38b1-42d7-a873-df2b0aef42eb" containerName="extract" Jul 17 18:16:44.676414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.675877 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6441300-acfd-4e7e-87bc-463e6baccaa3" containerName="extract" Jul 17 18:16:44.683844 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.683823 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:44.686950 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.686929 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hhgcn\"" Jul 17 18:16:44.699595 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.699572 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l"] Jul 17 18:16:44.746106 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.746068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zhk\" (UniqueName: \"kubernetes.io/projected/3490bbda-efb7-4dfb-b649-044cac665e15-kube-api-access-l6zhk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:44.746258 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.746116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3490bbda-efb7-4dfb-b649-044cac665e15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:44.846588 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.846551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3490bbda-efb7-4dfb-b649-044cac665e15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:44.846741 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.846627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zhk\" (UniqueName: \"kubernetes.io/projected/3490bbda-efb7-4dfb-b649-044cac665e15-kube-api-access-l6zhk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:44.846910 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.846889 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3490bbda-efb7-4dfb-b649-044cac665e15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:44.858311 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.858287 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zhk\" (UniqueName: \"kubernetes.io/projected/3490bbda-efb7-4dfb-b649-044cac665e15-kube-api-access-l6zhk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:44.993252 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:44.993213 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:45.131568 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:45.131537 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l"] Jul 17 18:16:45.133600 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:45.133568 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3490bbda_efb7_4dfb_b649_044cac665e15.slice/crio-8481d276bb0d723802a6ff5df3804e408df2924eb0ebc7653c232cfdf26398e9 WatchSource:0}: Error finding container 8481d276bb0d723802a6ff5df3804e408df2924eb0ebc7653c232cfdf26398e9: Status 404 returned error can't find the container with id 8481d276bb0d723802a6ff5df3804e408df2924eb0ebc7653c232cfdf26398e9 Jul 17 18:16:45.874036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:45.873995 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" event={"ID":"3490bbda-efb7-4dfb-b649-044cac665e15","Type":"ContainerStarted","Data":"8481d276bb0d723802a6ff5df3804e408df2924eb0ebc7653c232cfdf26398e9"} Jul 17 18:16:47.673539 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:47.673505 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns"] Jul 17 18:16:47.676964 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:47.676937 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:16:47.680890 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:16:47.680857 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-operator-75cc8d5cb7-nt6ns\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="03f56fa4-6812-4702-aa55-71d3c7814f5e" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:16:47.681044 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:16:47.680936 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"authorino-operator-dockercfg-hn96t\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hn96t\"" type="*v1.Secret" Jul 17 18:16:47.705762 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:47.705733 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns"] Jul 17 18:16:47.766359 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:47.766317 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsj2m\" (UniqueName: \"kubernetes.io/projected/03f56fa4-6812-4702-aa55-71d3c7814f5e-kube-api-access-xsj2m\") pod \"authorino-operator-75cc8d5cb7-nt6ns\" (UID: \"03f56fa4-6812-4702-aa55-71d3c7814f5e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:16:47.866979 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:47.866924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsj2m\" (UniqueName: \"kubernetes.io/projected/03f56fa4-6812-4702-aa55-71d3c7814f5e-kube-api-access-xsj2m\") pod \"authorino-operator-75cc8d5cb7-nt6ns\" (UID: \"03f56fa4-6812-4702-aa55-71d3c7814f5e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:16:47.877378 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:47.877353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsj2m\" (UniqueName: \"kubernetes.io/projected/03f56fa4-6812-4702-aa55-71d3c7814f5e-kube-api-access-xsj2m\") pod \"authorino-operator-75cc8d5cb7-nt6ns\" (UID: \"03f56fa4-6812-4702-aa55-71d3c7814f5e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:16:48.686252 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:48.686226 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hn96t\"" Jul 17 18:16:48.688309 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:48.688292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:16:48.833560 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:48.833527 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns"] Jul 17 18:16:48.834734 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:48.834696 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f56fa4_6812_4702_aa55_71d3c7814f5e.slice/crio-8fda32cf6c2b7205bc3066ab24bbf50072bfe04e63bc15a7d3f08bcf0a174dba WatchSource:0}: Error finding container 8fda32cf6c2b7205bc3066ab24bbf50072bfe04e63bc15a7d3f08bcf0a174dba: Status 404 returned error can't find the container with id 8fda32cf6c2b7205bc3066ab24bbf50072bfe04e63bc15a7d3f08bcf0a174dba Jul 17 18:16:48.891311 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:48.891281 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" event={"ID":"03f56fa4-6812-4702-aa55-71d3c7814f5e","Type":"ContainerStarted","Data":"8fda32cf6c2b7205bc3066ab24bbf50072bfe04e63bc15a7d3f08bcf0a174dba"} Jul 17 18:16:49.609010 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.608812 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm"] Jul 17 18:16:49.612434 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.612400 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" Jul 17 18:16:49.615457 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.615415 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 17 18:16:49.616622 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.616600 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qhwm9\"" Jul 17 18:16:49.637009 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.636907 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm"] Jul 17 18:16:49.679668 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.679633 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxx2\" (UniqueName: \"kubernetes.io/projected/bdc997a8-42fe-43da-8c59-40ed30a843b7-kube-api-access-vlxx2\") pod \"dns-operator-controller-manager-864dd78c4c-ftlmm\" (UID: \"bdc997a8-42fe-43da-8c59-40ed30a843b7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" Jul 17 18:16:49.780788 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.780749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxx2\" (UniqueName: \"kubernetes.io/projected/bdc997a8-42fe-43da-8c59-40ed30a843b7-kube-api-access-vlxx2\") pod \"dns-operator-controller-manager-864dd78c4c-ftlmm\" (UID: \"bdc997a8-42fe-43da-8c59-40ed30a843b7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" Jul 17 18:16:49.797299 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.797263 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxx2\" (UniqueName: \"kubernetes.io/projected/bdc997a8-42fe-43da-8c59-40ed30a843b7-kube-api-access-vlxx2\") pod \"dns-operator-controller-manager-864dd78c4c-ftlmm\" (UID: \"bdc997a8-42fe-43da-8c59-40ed30a843b7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" Jul 17 18:16:49.931124 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.931091 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qhwm9\"" Jul 17 18:16:49.939536 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:49.939503 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" Jul 17 18:16:50.131004 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:50.130903 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm"] Jul 17 18:16:51.251048 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:51.251014 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc997a8_42fe_43da_8c59_40ed30a843b7.slice/crio-a3883db9b958ee912abe291c1513007160adca10284fc31101053064f279eb61 WatchSource:0}: Error finding container a3883db9b958ee912abe291c1513007160adca10284fc31101053064f279eb61: Status 404 returned error can't find the container with id a3883db9b958ee912abe291c1513007160adca10284fc31101053064f279eb61 Jul 17 18:16:51.909032 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:51.908991 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" event={"ID":"bdc997a8-42fe-43da-8c59-40ed30a843b7","Type":"ContainerStarted","Data":"a3883db9b958ee912abe291c1513007160adca10284fc31101053064f279eb61"} Jul 17 18:16:52.302486 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.302461 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps"] Jul 17 18:16:52.307038 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.307013 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:16:52.310556 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.310531 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-858cr\"" Jul 17 18:16:52.328542 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.328517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps"] Jul 17 18:16:52.400023 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.399987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd287\" (UniqueName: \"kubernetes.io/projected/000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2-kube-api-access-qd287\") pod \"limitador-operator-controller-manager-859d7b78bd-dfhps\" (UID: \"000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:16:52.501112 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.501010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd287\" (UniqueName: \"kubernetes.io/projected/000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2-kube-api-access-qd287\") pod \"limitador-operator-controller-manager-859d7b78bd-dfhps\" (UID: \"000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:16:52.513875 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.513849 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd287\" (UniqueName: \"kubernetes.io/projected/000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2-kube-api-access-qd287\") pod \"limitador-operator-controller-manager-859d7b78bd-dfhps\" (UID: \"000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:16:52.626390 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.626351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:16:52.919796 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.919572 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" event={"ID":"3490bbda-efb7-4dfb-b649-044cac665e15","Type":"ContainerStarted","Data":"208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562"} Jul 17 18:16:52.920008 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.919813 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:16:52.922154 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.922124 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" event={"ID":"03f56fa4-6812-4702-aa55-71d3c7814f5e","Type":"ContainerStarted","Data":"690de7a091fe0502c3be5daee67ecdf30a75f8f030edbb08dab0cf6e6d91150f"} Jul 17 18:16:52.922297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.922269 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:16:52.941924 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.941872 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" podStartSLOduration=1.8167102480000001 podStartE2EDuration="8.941854888s" totalImagesPullingTime="7.12514464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:52.939361118 +0000 UTC m=+423.354099144" watchObservedRunningTime="2026-07-17 18:16:52.941854888 +0000 UTC m=+423.356592914" Jul 17 18:16:52.966149 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:52.966099 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" podStartSLOduration=2.535499644 podStartE2EDuration="5.966081626s" totalImagesPullingTime="3.430581982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:52.963439109 +0000 UTC m=+423.378177174" watchObservedRunningTime="2026-07-17 18:16:52.966081626 +0000 UTC m=+423.380819656" Jul 17 18:16:53.073373 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:53.073337 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps"] Jul 17 18:16:53.085443 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:16:53.085407 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000db9a5_60d5_4a8e_93b2_3fdb8ec5b4f2.slice/crio-8a8ee647f18780c40249b0e437dbe585cdce2bde87e7c684608168f997db527f WatchSource:0}: Error finding container 8a8ee647f18780c40249b0e437dbe585cdce2bde87e7c684608168f997db527f: Status 404 returned error can't find the container with id 8a8ee647f18780c40249b0e437dbe585cdce2bde87e7c684608168f997db527f Jul 17 18:16:53.930281 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:53.930242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" event={"ID":"000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2","Type":"ContainerStarted","Data":"8a8ee647f18780c40249b0e437dbe585cdce2bde87e7c684608168f997db527f"} Jul 17 18:16:54.098736 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:54.098651 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 17 18:16:54.937900 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:54.937853 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" event={"ID":"bdc997a8-42fe-43da-8c59-40ed30a843b7","Type":"ContainerStarted","Data":"3aedfca058cf9bf2b44db6c0a8f83b4dfccc17e7df58080b49b6f992953f529c"} Jul 17 18:16:54.938412 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:54.937944 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" Jul 17 18:16:55.945190 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:55.945145 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" event={"ID":"000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2","Type":"ContainerStarted","Data":"3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934"} Jul 17 18:16:55.974128 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:55.974074 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" podStartSLOduration=1.941777498 podStartE2EDuration="3.974058635s" totalImagesPullingTime="2.032281137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:55.973737234 +0000 UTC m=+426.388475275" watchObservedRunningTime="2026-07-17 18:16:55.974058635 +0000 UTC m=+426.388796662" Jul 17 18:16:55.975042 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:55.975014 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" podStartSLOduration=4.139358576 podStartE2EDuration="6.975004085s" totalImagesPullingTime="2.835645509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:54.983136599 +0000 UTC m=+425.397874626" watchObservedRunningTime="2026-07-17 18:16:55.975004085 +0000 UTC m=+426.389742108" Jul 17 18:16:56.950519 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:16:56.950492 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:03.933060 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:03.933027 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nt6ns" Jul 17 18:17:03.933505 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:03.933308 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:05.948044 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:05.948012 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ftlmm" Jul 17 18:17:06.054621 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.054584 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l"] Jul 17 18:17:06.054885 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.054837 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" containerName="manager" containerID="cri-o://208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562" gracePeriod=2 Jul 17 18:17:06.081458 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.081431 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l"] Jul 17 18:17:06.091577 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.091528 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3490bbda-efb7-4dfb-b649-044cac665e15\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\": pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.101215 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.101187 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps"] Jul 17 18:17:06.101391 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.101370 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" containerName="manager" containerID="cri-o://3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934" gracePeriod=2 Jul 17 18:17:06.103148 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.103124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:06.105380 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.105350 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.121092 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.121062 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps"] Jul 17 18:17:06.121521 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.121493 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.124840 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.124813 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf"] Jul 17 18:17:06.125663 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.125642 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" containerName="manager" Jul 17 18:17:06.125783 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.125773 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" containerName="manager" Jul 17 18:17:06.125900 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.125890 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" containerName="manager" Jul 17 18:17:06.125993 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.125984 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" containerName="manager" Jul 17 18:17:06.126232 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.126220 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" containerName="manager" Jul 17 18:17:06.126315 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.126306 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" containerName="manager" Jul 17 18:17:06.131217 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.131197 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.138637 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.138516 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.145072 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.145046 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn"] Jul 17 18:17:06.148838 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.148819 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" Jul 17 18:17:06.187362 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.187316 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dfhps\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:06.188530 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.188495 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn"] Jul 17 18:17:06.196341 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.196310 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.199456 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.199400 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dfhps\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:06.215507 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.215483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmzm\" (UniqueName: \"kubernetes.io/projected/7e5dd04e-25db-4ef2-9694-2e2f0993ffc3-kube-api-access-dtmzm\") pod \"limitador-operator-controller-manager-859d7b78bd-f7lfn\" (UID: \"7e5dd04e-25db-4ef2-9694-2e2f0993ffc3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" Jul 17 18:17:06.215572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.215540 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1171c7c-2761-41d3-84ba-880bd831614e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lphtf\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.215613 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.215583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npntk\" (UniqueName: \"kubernetes.io/projected/b1171c7c-2761-41d3-84ba-880bd831614e-kube-api-access-npntk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lphtf\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.247908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.247876 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf"] Jul 17 18:17:06.316782 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.316750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmzm\" (UniqueName: \"kubernetes.io/projected/7e5dd04e-25db-4ef2-9694-2e2f0993ffc3-kube-api-access-dtmzm\") pod \"limitador-operator-controller-manager-859d7b78bd-f7lfn\" (UID: \"7e5dd04e-25db-4ef2-9694-2e2f0993ffc3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" Jul 17 18:17:06.317004 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.316800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1171c7c-2761-41d3-84ba-880bd831614e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lphtf\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.317004 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.316837 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npntk\" (UniqueName: \"kubernetes.io/projected/b1171c7c-2761-41d3-84ba-880bd831614e-kube-api-access-npntk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lphtf\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.317299 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.317274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1171c7c-2761-41d3-84ba-880bd831614e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lphtf\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.362328 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.362308 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.365081 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.365062 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:06.366614 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.366578 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.366769 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.366752 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npntk\" (UniqueName: \"kubernetes.io/projected/b1171c7c-2761-41d3-84ba-880bd831614e-kube-api-access-npntk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lphtf\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.369332 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.369310 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dfhps\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:06.374787 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.374764 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:06.377782 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:06.377761 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dfhps\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:06.390587 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.390559 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmzm\" (UniqueName: \"kubernetes.io/projected/7e5dd04e-25db-4ef2-9694-2e2f0993ffc3-kube-api-access-dtmzm\") pod \"limitador-operator-controller-manager-859d7b78bd-f7lfn\" (UID: \"7e5dd04e-25db-4ef2-9694-2e2f0993ffc3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" Jul 17 18:17:06.417344 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.417319 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3490bbda-efb7-4dfb-b649-044cac665e15-extensions-socket-volume\") pod \"3490bbda-efb7-4dfb-b649-044cac665e15\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " Jul 17 18:17:06.417434 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.417387 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6zhk\" (UniqueName: \"kubernetes.io/projected/3490bbda-efb7-4dfb-b649-044cac665e15-kube-api-access-l6zhk\") pod \"3490bbda-efb7-4dfb-b649-044cac665e15\" (UID: \"3490bbda-efb7-4dfb-b649-044cac665e15\") " Jul 17 18:17:06.417434 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.417415 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd287\" (UniqueName: \"kubernetes.io/projected/000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2-kube-api-access-qd287\") pod \"000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2\" (UID: \"000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2\") " Jul 17 18:17:06.417825 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.417793 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3490bbda-efb7-4dfb-b649-044cac665e15-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3490bbda-efb7-4dfb-b649-044cac665e15" (UID: "3490bbda-efb7-4dfb-b649-044cac665e15"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:06.419342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.419314 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3490bbda-efb7-4dfb-b649-044cac665e15-kube-api-access-l6zhk" (OuterVolumeSpecName: "kube-api-access-l6zhk") pod "3490bbda-efb7-4dfb-b649-044cac665e15" (UID: "3490bbda-efb7-4dfb-b649-044cac665e15"). InnerVolumeSpecName "kube-api-access-l6zhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:06.419493 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.419475 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2-kube-api-access-qd287" (OuterVolumeSpecName: "kube-api-access-qd287") pod "000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" (UID: "000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2"). InnerVolumeSpecName "kube-api-access-qd287". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:06.501461 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.501374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:06.508870 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.508850 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" Jul 17 18:17:06.518314 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.518284 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd287\" (UniqueName: \"kubernetes.io/projected/000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2-kube-api-access-qd287\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:17:06.518314 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.518316 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3490bbda-efb7-4dfb-b649-044cac665e15-extensions-socket-volume\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:17:06.518509 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.518330 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zhk\" (UniqueName: \"kubernetes.io/projected/3490bbda-efb7-4dfb-b649-044cac665e15-kube-api-access-l6zhk\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:17:06.667123 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.667100 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf"] Jul 17 18:17:06.669049 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:17:06.669026 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1171c7c_2761_41d3_84ba_880bd831614e.slice/crio-e4d69e7eedc50996ff89b32c37bf515a57502c8fedbe92df0fbea0b50484c490 WatchSource:0}: Error finding container e4d69e7eedc50996ff89b32c37bf515a57502c8fedbe92df0fbea0b50484c490: Status 404 returned error can't find the container with id e4d69e7eedc50996ff89b32c37bf515a57502c8fedbe92df0fbea0b50484c490 Jul 17 18:17:06.691387 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:06.691366 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn"] Jul 17 18:17:06.699162 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:17:06.699136 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5dd04e_25db_4ef2_9694_2e2f0993ffc3.slice/crio-cc1febef7e83949ec8e34154586f64425349ff3376a37d7e8fe93efbed548e25 WatchSource:0}: Error finding container cc1febef7e83949ec8e34154586f64425349ff3376a37d7e8fe93efbed548e25: Status 404 returned error can't find the container with id cc1febef7e83949ec8e34154586f64425349ff3376a37d7e8fe93efbed548e25 Jul 17 18:17:07.006478 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.006442 2570 generic.go:356] "Generic (PLEG): container finished" podID="3490bbda-efb7-4dfb-b649-044cac665e15" containerID="208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562" exitCode=0 Jul 17 18:17:07.006975 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.006488 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:07.006975 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.006540 2570 scope.go:117] "RemoveContainer" containerID="208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562" Jul 17 18:17:07.008311 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.008282 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" event={"ID":"7e5dd04e-25db-4ef2-9694-2e2f0993ffc3","Type":"ContainerStarted","Data":"117e8ecd0248a90ffe0d0837ba0b4f9da4441841abb179a58ebb18e8abe7e5dd"} Jul 17 18:17:07.008432 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.008348 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" Jul 17 18:17:07.008432 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.008376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" event={"ID":"7e5dd04e-25db-4ef2-9694-2e2f0993ffc3","Type":"ContainerStarted","Data":"cc1febef7e83949ec8e34154586f64425349ff3376a37d7e8fe93efbed548e25"} Jul 17 18:17:07.010238 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.010217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" event={"ID":"b1171c7c-2761-41d3-84ba-880bd831614e","Type":"ContainerStarted","Data":"c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9"} Jul 17 18:17:07.010342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.010266 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" event={"ID":"b1171c7c-2761-41d3-84ba-880bd831614e","Type":"ContainerStarted","Data":"e4d69e7eedc50996ff89b32c37bf515a57502c8fedbe92df0fbea0b50484c490"} Jul 17 18:17:07.010342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.010301 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:07.010709 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.010681 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:07.011736 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.011662 2570 generic.go:356] "Generic (PLEG): container finished" podID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" containerID="3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934" exitCode=0 Jul 17 18:17:07.011736 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.011696 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:07.013537 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.013516 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dfhps\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:07.017551 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.017522 2570 scope.go:117] "RemoveContainer" containerID="208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562" Jul 17 18:17:07.017684 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.017657 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dfhps\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:07.017830 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.017812 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562\": container with ID starting with 208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562 not found: ID does not exist" containerID="208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562" Jul 17 18:17:07.017874 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.017839 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562"} err="failed to get container status \"208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562\": rpc error: code = NotFound desc = could not find container \"208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562\": container with ID starting with 208b1f75d75259615734db2b7b3601e7f3a1d84253ad4bd815a9dab86b558562 not found: ID does not exist" Jul 17 18:17:07.017874 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.017855 2570 scope.go:117] "RemoveContainer" containerID="3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934" Jul 17 18:17:07.027127 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.027105 2570 scope.go:117] "RemoveContainer" containerID="3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934" Jul 17 18:17:07.027439 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.027423 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934\": container with ID starting with 3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934 not found: ID does not exist" containerID="3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934" Jul 17 18:17:07.027504 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.027448 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934"} err="failed to get container status \"3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934\": rpc error: code = NotFound desc = could not find container \"3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934\": container with ID starting with 3fa83cdeb263180b236cd078d7657e7dd4185cbe3d081e963e4e16548c85e934 not found: ID does not exist" Jul 17 18:17:07.054436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.054383 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" podStartSLOduration=1.0543678 podStartE2EDuration="1.0543678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:07.052866244 +0000 UTC m=+437.467604275" watchObservedRunningTime="2026-07-17 18:17:07.0543678 +0000 UTC m=+437.469105824" Jul 17 18:17:07.055393 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.055370 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:07.114893 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:07.114829 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" podStartSLOduration=1.114810983 podStartE2EDuration="1.114810983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:07.113690348 +0000 UTC m=+437.528428374" watchObservedRunningTime="2026-07-17 18:17:07.114810983 +0000 UTC m=+437.529549053" Jul 17 18:17:07.118431 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.118390 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5pl7l\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5pl7l" Jul 17 18:17:07.120780 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:07.120748 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dfhps\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dfhps" Jul 17 18:17:08.072612 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:08.072581 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2" path="/var/lib/kubelet/pods/000db9a5-60d5-4a8e-93b2-3fdb8ec5b4f2/volumes" Jul 17 18:17:08.073194 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:08.072913 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3490bbda-efb7-4dfb-b649-044cac665e15" path="/var/lib/kubelet/pods/3490bbda-efb7-4dfb-b649-044cac665e15/volumes" Jul 17 18:17:09.756681 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.756648 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc"] Jul 17 18:17:09.761023 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.761004 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:09.793019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.792982 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc"] Jul 17 18:17:09.844149 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.844105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkt8p\" (UniqueName: \"kubernetes.io/projected/588d7f5c-a88c-4039-8ba1-1b6f13536835-kube-api-access-qkt8p\") pod \"kuadrant-operator-controller-manager-86878c769f-wsbpc\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:09.844330 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.844229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588d7f5c-a88c-4039-8ba1-1b6f13536835-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wsbpc\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:09.945674 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.945635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt8p\" (UniqueName: \"kubernetes.io/projected/588d7f5c-a88c-4039-8ba1-1b6f13536835-kube-api-access-qkt8p\") pod \"kuadrant-operator-controller-manager-86878c769f-wsbpc\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:09.945855 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.945710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588d7f5c-a88c-4039-8ba1-1b6f13536835-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wsbpc\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:09.946112 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.946096 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588d7f5c-a88c-4039-8ba1-1b6f13536835-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wsbpc\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:09.976836 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:09.976807 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt8p\" (UniqueName: \"kubernetes.io/projected/588d7f5c-a88c-4039-8ba1-1b6f13536835-kube-api-access-qkt8p\") pod \"kuadrant-operator-controller-manager-86878c769f-wsbpc\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:10.070927 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:10.070838 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:10.261475 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:10.261451 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc"] Jul 17 18:17:10.264364 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:17:10.264332 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588d7f5c_a88c_4039_8ba1_1b6f13536835.slice/crio-455d3e201eeae3fc8c25c014d9122d20cd8ea96ce3d0425cea069be374643ff0 WatchSource:0}: Error finding container 455d3e201eeae3fc8c25c014d9122d20cd8ea96ce3d0425cea069be374643ff0: Status 404 returned error can't find the container with id 455d3e201eeae3fc8c25c014d9122d20cd8ea96ce3d0425cea069be374643ff0 Jul 17 18:17:11.036696 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:11.036665 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" event={"ID":"588d7f5c-a88c-4039-8ba1-1b6f13536835","Type":"ContainerStarted","Data":"aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69"} Jul 17 18:17:11.037208 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:11.036713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" event={"ID":"588d7f5c-a88c-4039-8ba1-1b6f13536835","Type":"ContainerStarted","Data":"455d3e201eeae3fc8c25c014d9122d20cd8ea96ce3d0425cea069be374643ff0"} Jul 17 18:17:11.037208 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:11.036805 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:11.097031 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:11.096978 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" podStartSLOduration=2.096947065 podStartE2EDuration="2.096947065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:11.095607765 +0000 UTC m=+441.510345805" watchObservedRunningTime="2026-07-17 18:17:11.096947065 +0000 UTC m=+441.511685092" Jul 17 18:17:18.020620 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:18.020585 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f7lfn" Jul 17 18:17:18.021051 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:18.020722 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:22.045033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.045001 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:17:22.135817 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.135787 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf"] Jul 17 18:17:22.136037 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.136012 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" podUID="b1171c7c-2761-41d3-84ba-880bd831614e" containerName="manager" containerID="cri-o://c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9" gracePeriod=10 Jul 17 18:17:22.384654 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.384630 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:22.461345 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.461313 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npntk\" (UniqueName: \"kubernetes.io/projected/b1171c7c-2761-41d3-84ba-880bd831614e-kube-api-access-npntk\") pod \"b1171c7c-2761-41d3-84ba-880bd831614e\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " Jul 17 18:17:22.461503 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.461409 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1171c7c-2761-41d3-84ba-880bd831614e-extensions-socket-volume\") pod \"b1171c7c-2761-41d3-84ba-880bd831614e\" (UID: \"b1171c7c-2761-41d3-84ba-880bd831614e\") " Jul 17 18:17:22.461823 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.461801 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1171c7c-2761-41d3-84ba-880bd831614e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b1171c7c-2761-41d3-84ba-880bd831614e" (UID: "b1171c7c-2761-41d3-84ba-880bd831614e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:22.463433 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.463412 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1171c7c-2761-41d3-84ba-880bd831614e-kube-api-access-npntk" (OuterVolumeSpecName: "kube-api-access-npntk") pod "b1171c7c-2761-41d3-84ba-880bd831614e" (UID: "b1171c7c-2761-41d3-84ba-880bd831614e"). InnerVolumeSpecName "kube-api-access-npntk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:22.562179 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.562101 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1171c7c-2761-41d3-84ba-880bd831614e-extensions-socket-volume\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:17:22.562179 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:22.562128 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npntk\" (UniqueName: \"kubernetes.io/projected/b1171c7c-2761-41d3-84ba-880bd831614e-kube-api-access-npntk\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:17:23.098532 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.098501 2570 generic.go:356] "Generic (PLEG): container finished" podID="b1171c7c-2761-41d3-84ba-880bd831614e" containerID="c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9" exitCode=0 Jul 17 18:17:23.099011 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.098539 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" event={"ID":"b1171c7c-2761-41d3-84ba-880bd831614e","Type":"ContainerDied","Data":"c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9"} Jul 17 18:17:23.099011 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.098544 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" Jul 17 18:17:23.099011 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.098577 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf" event={"ID":"b1171c7c-2761-41d3-84ba-880bd831614e","Type":"ContainerDied","Data":"e4d69e7eedc50996ff89b32c37bf515a57502c8fedbe92df0fbea0b50484c490"} Jul 17 18:17:23.099011 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.098597 2570 scope.go:117] "RemoveContainer" containerID="c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9" Jul 17 18:17:23.108106 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.108085 2570 scope.go:117] "RemoveContainer" containerID="c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9" Jul 17 18:17:23.108346 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:23.108328 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9\": container with ID starting with c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9 not found: ID does not exist" containerID="c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9" Jul 17 18:17:23.108407 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.108357 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9"} err="failed to get container status \"c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9\": rpc error: code = NotFound desc = could not find container \"c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9\": container with ID starting with c9d76b62215afc9cdcf3cd35a21773ef46f42bd43c420bbc212c73d00691edb9 not found: ID does not exist" Jul 17 18:17:23.145216 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.145171 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf"] Jul 17 18:17:23.165629 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:23.165598 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lphtf"] Jul 17 18:17:24.072687 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:24.072652 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1171c7c-2761-41d3-84ba-880bd831614e" path="/var/lib/kubelet/pods/b1171c7c-2761-41d3-84ba-880bd831614e/volumes" Jul 17 18:17:42.667984 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.667921 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kmqpc"] Jul 17 18:17:42.668464 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.668399 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1171c7c-2761-41d3-84ba-880bd831614e" containerName="manager" Jul 17 18:17:42.668464 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.668417 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1171c7c-2761-41d3-84ba-880bd831614e" containerName="manager" Jul 17 18:17:42.668579 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.668510 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1171c7c-2761-41d3-84ba-880bd831614e" containerName="manager" Jul 17 18:17:42.670337 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.670316 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:42.674376 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.674356 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 18:17:42.674497 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.674482 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zgbg7\"" Jul 17 18:17:42.722515 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.722482 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kmqpc"] Jul 17 18:17:42.807032 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.807002 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kmqpc"] Jul 17 18:17:42.831098 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.831062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zww4\" (UniqueName: \"kubernetes.io/projected/338645a5-c9f0-461e-9def-039012879920-kube-api-access-4zww4\") pod \"limitador-limitador-64c94fd64f-kmqpc\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:42.831271 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.831104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/338645a5-c9f0-461e-9def-039012879920-config-file\") pod \"limitador-limitador-64c94fd64f-kmqpc\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:42.931539 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.931508 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zww4\" (UniqueName: \"kubernetes.io/projected/338645a5-c9f0-461e-9def-039012879920-kube-api-access-4zww4\") pod \"limitador-limitador-64c94fd64f-kmqpc\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:42.931693 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.931553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/338645a5-c9f0-461e-9def-039012879920-config-file\") pod \"limitador-limitador-64c94fd64f-kmqpc\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:42.932200 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.932182 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/338645a5-c9f0-461e-9def-039012879920-config-file\") pod \"limitador-limitador-64c94fd64f-kmqpc\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:42.946592 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.946554 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zww4\" (UniqueName: \"kubernetes.io/projected/338645a5-c9f0-461e-9def-039012879920-kube-api-access-4zww4\") pod \"limitador-limitador-64c94fd64f-kmqpc\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:42.980158 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:42.980120 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:43.176713 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:17:43.176682 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338645a5_c9f0_461e_9def_039012879920.slice/crio-7e7388b617f196c9e1f9fe1f2d32aa22ce6406b5a5674b4e0d152edf049ef832 WatchSource:0}: Error finding container 7e7388b617f196c9e1f9fe1f2d32aa22ce6406b5a5674b4e0d152edf049ef832: Status 404 returned error can't find the container with id 7e7388b617f196c9e1f9fe1f2d32aa22ce6406b5a5674b4e0d152edf049ef832 Jul 17 18:17:43.176829 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:43.176719 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kmqpc"] Jul 17 18:17:43.199903 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:43.199823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" event={"ID":"338645a5-c9f0-461e-9def-039012879920","Type":"ContainerStarted","Data":"7e7388b617f196c9e1f9fe1f2d32aa22ce6406b5a5674b4e0d152edf049ef832"} Jul 17 18:17:44.170157 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.170115 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-bw6f6"] Jul 17 18:17:44.173035 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.173009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bw6f6" Jul 17 18:17:44.179133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.178862 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-88ddz\"" Jul 17 18:17:44.251921 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.251850 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bw6f6"] Jul 17 18:17:44.344821 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.344687 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsgc\" (UniqueName: \"kubernetes.io/projected/b3f3f532-696f-47f8-ab2c-e661962213ad-kube-api-access-2xsgc\") pod \"authorino-75d556bd77-bw6f6\" (UID: \"b3f3f532-696f-47f8-ab2c-e661962213ad\") " pod="kuadrant-system/authorino-75d556bd77-bw6f6" Jul 17 18:17:44.446423 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.446369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsgc\" (UniqueName: \"kubernetes.io/projected/b3f3f532-696f-47f8-ab2c-e661962213ad-kube-api-access-2xsgc\") pod \"authorino-75d556bd77-bw6f6\" (UID: \"b3f3f532-696f-47f8-ab2c-e661962213ad\") " pod="kuadrant-system/authorino-75d556bd77-bw6f6" Jul 17 18:17:44.471258 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.470921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsgc\" (UniqueName: \"kubernetes.io/projected/b3f3f532-696f-47f8-ab2c-e661962213ad-kube-api-access-2xsgc\") pod \"authorino-75d556bd77-bw6f6\" (UID: \"b3f3f532-696f-47f8-ab2c-e661962213ad\") " pod="kuadrant-system/authorino-75d556bd77-bw6f6" Jul 17 18:17:44.486843 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.486810 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bw6f6" Jul 17 18:17:44.719294 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:17:44.719257 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3f3f532_696f_47f8_ab2c_e661962213ad.slice/crio-f9678a74bbd1e46c4a8bec67f24de1326803a35ca4cfdeeac9f10055e9913200 WatchSource:0}: Error finding container f9678a74bbd1e46c4a8bec67f24de1326803a35ca4cfdeeac9f10055e9913200: Status 404 returned error can't find the container with id f9678a74bbd1e46c4a8bec67f24de1326803a35ca4cfdeeac9f10055e9913200 Jul 17 18:17:44.725704 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:44.725672 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bw6f6"] Jul 17 18:17:45.213911 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:45.213875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bw6f6" event={"ID":"b3f3f532-696f-47f8-ab2c-e661962213ad","Type":"ContainerStarted","Data":"f9678a74bbd1e46c4a8bec67f24de1326803a35ca4cfdeeac9f10055e9913200"} Jul 17 18:17:47.229721 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:47.229679 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" event={"ID":"338645a5-c9f0-461e-9def-039012879920","Type":"ContainerStarted","Data":"7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40"} Jul 17 18:17:47.230280 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:47.229835 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:47.273310 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:47.273248 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" podStartSLOduration=2.195437398 podStartE2EDuration="5.273224878s" totalImagesPullingTime="3.07778748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:47.267769599 +0000 UTC m=+477.682507660" watchObservedRunningTime="2026-07-17 18:17:47.273224878 +0000 UTC m=+477.687962905" Jul 17 18:17:49.242540 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:49.242505 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bw6f6" event={"ID":"b3f3f532-696f-47f8-ab2c-e661962213ad","Type":"ContainerStarted","Data":"e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207"} Jul 17 18:17:49.268333 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:49.268280 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-bw6f6" podStartSLOduration=1.595667403 podStartE2EDuration="5.268262009s" totalImagesPullingTime="3.672594606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:49.267163187 +0000 UTC m=+479.681901211" watchObservedRunningTime="2026-07-17 18:17:49.268262009 +0000 UTC m=+479.683000036" Jul 17 18:17:58.097521 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.097487 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kmqpc"] Jul 17 18:17:58.098123 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.097738 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" podUID="338645a5-c9f0-461e-9def-039012879920" containerName="limitador" containerID="cri-o://7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40" gracePeriod=30 Jul 17 18:17:58.098506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.098375 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:58.643772 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.643747 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:58.661527 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.661461 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zww4\" (UniqueName: \"kubernetes.io/projected/338645a5-c9f0-461e-9def-039012879920-kube-api-access-4zww4\") pod \"338645a5-c9f0-461e-9def-039012879920\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " Jul 17 18:17:58.661527 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.661502 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/338645a5-c9f0-461e-9def-039012879920-config-file\") pod \"338645a5-c9f0-461e-9def-039012879920\" (UID: \"338645a5-c9f0-461e-9def-039012879920\") " Jul 17 18:17:58.661914 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.661890 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338645a5-c9f0-461e-9def-039012879920-config-file" (OuterVolumeSpecName: "config-file") pod "338645a5-c9f0-461e-9def-039012879920" (UID: "338645a5-c9f0-461e-9def-039012879920"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:17:58.663554 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.663516 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338645a5-c9f0-461e-9def-039012879920-kube-api-access-4zww4" (OuterVolumeSpecName: "kube-api-access-4zww4") pod "338645a5-c9f0-461e-9def-039012879920" (UID: "338645a5-c9f0-461e-9def-039012879920"). InnerVolumeSpecName "kube-api-access-4zww4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:58.762297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.762260 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zww4\" (UniqueName: \"kubernetes.io/projected/338645a5-c9f0-461e-9def-039012879920-kube-api-access-4zww4\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:17:58.762297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:58.762291 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/338645a5-c9f0-461e-9def-039012879920-config-file\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:17:59.292483 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.292448 2570 generic.go:356] "Generic (PLEG): container finished" podID="338645a5-c9f0-461e-9def-039012879920" containerID="7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40" exitCode=0 Jul 17 18:17:59.292483 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.292485 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" event={"ID":"338645a5-c9f0-461e-9def-039012879920","Type":"ContainerDied","Data":"7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40"} Jul 17 18:17:59.293016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.292494 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" Jul 17 18:17:59.293016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.292521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kmqpc" event={"ID":"338645a5-c9f0-461e-9def-039012879920","Type":"ContainerDied","Data":"7e7388b617f196c9e1f9fe1f2d32aa22ce6406b5a5674b4e0d152edf049ef832"} Jul 17 18:17:59.293016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.292542 2570 scope.go:117] "RemoveContainer" containerID="7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40" Jul 17 18:17:59.301202 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.301185 2570 scope.go:117] "RemoveContainer" containerID="7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40" Jul 17 18:17:59.301451 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:17:59.301435 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40\": container with ID starting with 7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40 not found: ID does not exist" containerID="7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40" Jul 17 18:17:59.301502 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.301458 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40"} err="failed to get container status \"7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40\": rpc error: code = NotFound desc = could not find container \"7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40\": container with ID starting with 7749708d71cfa0e32e0e94d55a942b8a6fadc62d77b419f52c2d77142dd48b40 not found: ID does not exist" Jul 17 18:17:59.320195 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.320166 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kmqpc"] Jul 17 18:17:59.327138 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:17:59.327110 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kmqpc"] Jul 17 18:18:00.072301 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:00.072267 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="338645a5-c9f0-461e-9def-039012879920" path="/var/lib/kubelet/pods/338645a5-c9f0-461e-9def-039012879920/volumes" Jul 17 18:18:04.382493 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.382455 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd"] Jul 17 18:18:04.382940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.382924 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="338645a5-c9f0-461e-9def-039012879920" containerName="limitador" Jul 17 18:18:04.383012 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.382943 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="338645a5-c9f0-461e-9def-039012879920" containerName="limitador" Jul 17 18:18:04.383080 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.383068 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="338645a5-c9f0-461e-9def-039012879920" containerName="limitador" Jul 17 18:18:04.387475 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.387455 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.390269 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.390251 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xccz4\"" Jul 17 18:18:04.390699 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.390682 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:18:04.390750 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.390703 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 17 18:18:04.395041 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.395023 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 18:18:04.402405 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.402383 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rh77\" (UniqueName: \"kubernetes.io/projected/acb353a2-7707-401e-8205-9b50e8706be8-kube-api-access-5rh77\") pod \"postgres-69fb4594cf-ns9zd\" (UID: \"acb353a2-7707-401e-8205-9b50e8706be8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.402498 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.402435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/acb353a2-7707-401e-8205-9b50e8706be8-data\") pod \"postgres-69fb4594cf-ns9zd\" (UID: \"acb353a2-7707-401e-8205-9b50e8706be8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.425401 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.425361 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd"] Jul 17 18:18:04.503523 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.503478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/acb353a2-7707-401e-8205-9b50e8706be8-data\") pod \"postgres-69fb4594cf-ns9zd\" (UID: \"acb353a2-7707-401e-8205-9b50e8706be8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.503717 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.503557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rh77\" (UniqueName: \"kubernetes.io/projected/acb353a2-7707-401e-8205-9b50e8706be8-kube-api-access-5rh77\") pod \"postgres-69fb4594cf-ns9zd\" (UID: \"acb353a2-7707-401e-8205-9b50e8706be8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.503896 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.503867 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/acb353a2-7707-401e-8205-9b50e8706be8-data\") pod \"postgres-69fb4594cf-ns9zd\" (UID: \"acb353a2-7707-401e-8205-9b50e8706be8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.535239 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.535211 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rh77\" (UniqueName: \"kubernetes.io/projected/acb353a2-7707-401e-8205-9b50e8706be8-kube-api-access-5rh77\") pod \"postgres-69fb4594cf-ns9zd\" (UID: \"acb353a2-7707-401e-8205-9b50e8706be8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.698094 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.698063 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:04.836780 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:04.836757 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd"] Jul 17 18:18:04.837916 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:18:04.837894 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb353a2_7707_401e_8205_9b50e8706be8.slice/crio-b267616e0c5a39e2f35f0b2a2886eed3651eb355393849347f361e9767438e4b WatchSource:0}: Error finding container b267616e0c5a39e2f35f0b2a2886eed3651eb355393849347f361e9767438e4b: Status 404 returned error can't find the container with id b267616e0c5a39e2f35f0b2a2886eed3651eb355393849347f361e9767438e4b Jul 17 18:18:05.323508 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:05.323469 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" event={"ID":"acb353a2-7707-401e-8205-9b50e8706be8","Type":"ContainerStarted","Data":"b267616e0c5a39e2f35f0b2a2886eed3651eb355393849347f361e9767438e4b"} Jul 17 18:18:10.352651 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:10.352613 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" event={"ID":"acb353a2-7707-401e-8205-9b50e8706be8","Type":"ContainerStarted","Data":"479cb13cbf88665bebad20d2acd767c8b558e0a8138ed5579617739d84699aee"} Jul 17 18:18:10.353157 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:10.352774 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:10.371509 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:10.371457 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" podStartSLOduration=1.715384086 podStartE2EDuration="6.371443159s" totalImagesPullingTime="4.656059073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:18:10.369054526 +0000 UTC m=+500.783792554" watchObservedRunningTime="2026-07-17 18:18:10.371443159 +0000 UTC m=+500.786181186" Jul 17 18:18:16.393695 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:16.393666 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ns9zd" Jul 17 18:18:17.360489 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.360453 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7ff649c79d-njsgv"] Jul 17 18:18:17.370336 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.370306 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.373353 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.373329 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 17 18:18:17.378317 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.378293 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff649c79d-njsgv"] Jul 17 18:18:17.414362 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.414333 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91b11c87-68ae-4976-9f1b-2ec738ce1b37-tls-cert\") pod \"authorino-7ff649c79d-njsgv\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.414740 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.414368 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspmh\" (UniqueName: \"kubernetes.io/projected/91b11c87-68ae-4976-9f1b-2ec738ce1b37-kube-api-access-mspmh\") pod \"authorino-7ff649c79d-njsgv\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.514947 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.514907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91b11c87-68ae-4976-9f1b-2ec738ce1b37-tls-cert\") pod \"authorino-7ff649c79d-njsgv\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.514947 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.514975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspmh\" (UniqueName: \"kubernetes.io/projected/91b11c87-68ae-4976-9f1b-2ec738ce1b37-kube-api-access-mspmh\") pod \"authorino-7ff649c79d-njsgv\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.517485 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.517453 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91b11c87-68ae-4976-9f1b-2ec738ce1b37-tls-cert\") pod \"authorino-7ff649c79d-njsgv\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.524569 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.524546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspmh\" (UniqueName: \"kubernetes.io/projected/91b11c87-68ae-4976-9f1b-2ec738ce1b37-kube-api-access-mspmh\") pod \"authorino-7ff649c79d-njsgv\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.680646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.680613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:18:17.842706 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:17.842681 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff649c79d-njsgv"] Jul 17 18:18:17.844755 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:18:17.844724 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91b11c87_68ae_4976_9f1b_2ec738ce1b37.slice/crio-3cd54c0499edf2737bbbbd81ab98a6b3eefffeb3ac36883f4f9849a9b7aa3fa5 WatchSource:0}: Error finding container 3cd54c0499edf2737bbbbd81ab98a6b3eefffeb3ac36883f4f9849a9b7aa3fa5: Status 404 returned error can't find the container with id 3cd54c0499edf2737bbbbd81ab98a6b3eefffeb3ac36883f4f9849a9b7aa3fa5 Jul 17 18:18:18.399092 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.399051 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff649c79d-njsgv" event={"ID":"91b11c87-68ae-4976-9f1b-2ec738ce1b37","Type":"ContainerStarted","Data":"b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6"} Jul 17 18:18:18.399092 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.399093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff649c79d-njsgv" event={"ID":"91b11c87-68ae-4976-9f1b-2ec738ce1b37","Type":"ContainerStarted","Data":"3cd54c0499edf2737bbbbd81ab98a6b3eefffeb3ac36883f4f9849a9b7aa3fa5"} Jul 17 18:18:18.417871 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.417822 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7ff649c79d-njsgv" podStartSLOduration=1.038238067 podStartE2EDuration="1.417806676s" totalImagesPullingTime="379.568609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:18:18.416129542 +0000 UTC m=+508.830867596" watchObservedRunningTime="2026-07-17 18:18:18.417806676 +0000 UTC m=+508.832544703" Jul 17 18:18:18.449056 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.449028 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bw6f6"] Jul 17 18:18:18.449217 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.449196 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-bw6f6" podUID="b3f3f532-696f-47f8-ab2c-e661962213ad" containerName="authorino" containerID="cri-o://e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207" gracePeriod=30 Jul 17 18:18:18.698576 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.698551 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bw6f6" Jul 17 18:18:18.725719 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.725609 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xsgc\" (UniqueName: \"kubernetes.io/projected/b3f3f532-696f-47f8-ab2c-e661962213ad-kube-api-access-2xsgc\") pod \"b3f3f532-696f-47f8-ab2c-e661962213ad\" (UID: \"b3f3f532-696f-47f8-ab2c-e661962213ad\") " Jul 17 18:18:18.728127 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.728096 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f3f532-696f-47f8-ab2c-e661962213ad-kube-api-access-2xsgc" (OuterVolumeSpecName: "kube-api-access-2xsgc") pod "b3f3f532-696f-47f8-ab2c-e661962213ad" (UID: "b3f3f532-696f-47f8-ab2c-e661962213ad"). InnerVolumeSpecName "kube-api-access-2xsgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:18:18.826858 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:18.826778 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsgc\" (UniqueName: \"kubernetes.io/projected/b3f3f532-696f-47f8-ab2c-e661962213ad-kube-api-access-2xsgc\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:18:19.405123 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.405086 2570 generic.go:356] "Generic (PLEG): container finished" podID="b3f3f532-696f-47f8-ab2c-e661962213ad" containerID="e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207" exitCode=0 Jul 17 18:18:19.405123 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.405127 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-bw6f6" Jul 17 18:18:19.405387 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.405173 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bw6f6" event={"ID":"b3f3f532-696f-47f8-ab2c-e661962213ad","Type":"ContainerDied","Data":"e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207"} Jul 17 18:18:19.405387 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.405209 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-bw6f6" event={"ID":"b3f3f532-696f-47f8-ab2c-e661962213ad","Type":"ContainerDied","Data":"f9678a74bbd1e46c4a8bec67f24de1326803a35ca4cfdeeac9f10055e9913200"} Jul 17 18:18:19.405387 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.405229 2570 scope.go:117] "RemoveContainer" containerID="e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207" Jul 17 18:18:19.414278 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.414260 2570 scope.go:117] "RemoveContainer" containerID="e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207" Jul 17 18:18:19.414547 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:18:19.414530 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207\": container with ID starting with e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207 not found: ID does not exist" containerID="e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207" Jul 17 18:18:19.414627 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.414560 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207"} err="failed to get container status \"e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207\": rpc error: code = NotFound desc = could not find container \"e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207\": container with ID starting with e8ee22b2f2de93ebf0c6fd8369ca04a54ef61fa7b2567ac916744421b7c63207 not found: ID does not exist" Jul 17 18:18:19.426202 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.426172 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bw6f6"] Jul 17 18:18:19.429897 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:19.429873 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-bw6f6"] Jul 17 18:18:20.077330 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:20.077300 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f3f532-696f-47f8-ab2c-e661962213ad" path="/var/lib/kubelet/pods/b3f3f532-696f-47f8-ab2c-e661962213ad/volumes" Jul 17 18:18:59.992242 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:59.992204 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-4kd6z"] Jul 17 18:18:59.992726 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:59.992546 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f3f532-696f-47f8-ab2c-e661962213ad" containerName="authorino" Jul 17 18:18:59.992726 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:59.992557 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f3f532-696f-47f8-ab2c-e661962213ad" containerName="authorino" Jul 17 18:18:59.992726 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:59.992622 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f3f532-696f-47f8-ab2c-e661962213ad" containerName="authorino" Jul 17 18:18:59.996823 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:59.996802 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:18:59.999692 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:59.999672 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-785wp\"" Jul 17 18:18:59.999815 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:18:59.999667 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 17 18:19:00.007201 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.007168 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-4kd6z"] Jul 17 18:19:00.073561 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.073526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663d5575-ebca-4529-a9ef-9bbbd6137773-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-4kd6z\" (UID: \"663d5575-ebca-4529-a9ef-9bbbd6137773\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:00.073731 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.073577 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghvj2\" (UniqueName: \"kubernetes.io/projected/663d5575-ebca-4529-a9ef-9bbbd6137773-kube-api-access-ghvj2\") pod \"payload-pre-processing-7677b774bb-4kd6z\" (UID: \"663d5575-ebca-4529-a9ef-9bbbd6137773\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:00.174062 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.174029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663d5575-ebca-4529-a9ef-9bbbd6137773-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-4kd6z\" (UID: \"663d5575-ebca-4529-a9ef-9bbbd6137773\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:00.174241 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.174092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvj2\" (UniqueName: \"kubernetes.io/projected/663d5575-ebca-4529-a9ef-9bbbd6137773-kube-api-access-ghvj2\") pod \"payload-pre-processing-7677b774bb-4kd6z\" (UID: \"663d5575-ebca-4529-a9ef-9bbbd6137773\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:00.174701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.174680 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663d5575-ebca-4529-a9ef-9bbbd6137773-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-4kd6z\" (UID: \"663d5575-ebca-4529-a9ef-9bbbd6137773\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:00.187340 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.187311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvj2\" (UniqueName: \"kubernetes.io/projected/663d5575-ebca-4529-a9ef-9bbbd6137773-kube-api-access-ghvj2\") pod \"payload-pre-processing-7677b774bb-4kd6z\" (UID: \"663d5575-ebca-4529-a9ef-9bbbd6137773\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:00.309078 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.308979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:00.450304 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.450268 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-4kd6z"] Jul 17 18:19:00.451251 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:19:00.451227 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663d5575_ebca_4529_a9ef_9bbbd6137773.slice/crio-7a3eae6434113232c404616da415b4e21adb6350bfd1936032999d54bd12b39c WatchSource:0}: Error finding container 7a3eae6434113232c404616da415b4e21adb6350bfd1936032999d54bd12b39c: Status 404 returned error can't find the container with id 7a3eae6434113232c404616da415b4e21adb6350bfd1936032999d54bd12b39c Jul 17 18:19:00.620819 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:00.620727 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" event={"ID":"663d5575-ebca-4529-a9ef-9bbbd6137773","Type":"ContainerStarted","Data":"7a3eae6434113232c404616da415b4e21adb6350bfd1936032999d54bd12b39c"} Jul 17 18:19:02.634584 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:02.634545 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" event={"ID":"663d5575-ebca-4529-a9ef-9bbbd6137773","Type":"ContainerStarted","Data":"8ae9a892ea6d00055b852d4581737fc3aa35f5e55a02f96b14707db10bb24ffb"} Jul 17 18:19:02.635064 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:02.634603 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:02.652823 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:02.652770 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" podStartSLOduration=2.170695745 podStartE2EDuration="3.652752328s" totalImagesPullingTime="1.482056583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:19:02.651273379 +0000 UTC m=+553.066011407" watchObservedRunningTime="2026-07-17 18:19:02.652752328 +0000 UTC m=+553.067490355" Jul 17 18:19:13.642410 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:13.642376 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-4kd6z" Jul 17 18:19:17.497513 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.497475 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9"] Jul 17 18:19:17.502211 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.502194 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.505156 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.505134 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rp4s8\"" Jul 17 18:19:17.505302 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.505135 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 18:19:17.506191 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.506170 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 17 18:19:17.506316 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.506204 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 18:19:17.520903 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.520873 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9"] Jul 17 18:19:17.620799 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.620765 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.620799 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.620802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.621048 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.620829 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79d55\" (UniqueName: \"kubernetes.io/projected/4953ecfb-aa53-46cf-a313-481b07f060b2-kube-api-access-79d55\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.621048 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.620876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4953ecfb-aa53-46cf-a313-481b07f060b2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.621048 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.620905 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.621048 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.620923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.721646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.721612 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.721646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.721647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.721873 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.721712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.721873 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.721727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.721873 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.721750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79d55\" (UniqueName: \"kubernetes.io/projected/4953ecfb-aa53-46cf-a313-481b07f060b2-kube-api-access-79d55\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.721873 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.721775 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4953ecfb-aa53-46cf-a313-481b07f060b2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.722195 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.722162 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.722195 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.722189 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.722326 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.722212 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.724100 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.724078 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4953ecfb-aa53-46cf-a313-481b07f060b2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.724213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.724195 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4953ecfb-aa53-46cf-a313-481b07f060b2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.731090 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.731056 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79d55\" (UniqueName: \"kubernetes.io/projected/4953ecfb-aa53-46cf-a313-481b07f060b2-kube-api-access-79d55\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9\" (UID: \"4953ecfb-aa53-46cf-a313-481b07f060b2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.813724 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.813636 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:17.979193 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:17.979162 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9"] Jul 17 18:19:17.980650 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:19:17.980625 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4953ecfb_aa53_46cf_a313_481b07f060b2.slice/crio-252e96ea48885fc5433b828f3abb4d3bc558e1049247366041137ae10e31d5b1 WatchSource:0}: Error finding container 252e96ea48885fc5433b828f3abb4d3bc558e1049247366041137ae10e31d5b1: Status 404 returned error can't find the container with id 252e96ea48885fc5433b828f3abb4d3bc558e1049247366041137ae10e31d5b1 Jul 17 18:19:18.720896 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:18.720857 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" event={"ID":"4953ecfb-aa53-46cf-a313-481b07f060b2","Type":"ContainerStarted","Data":"252e96ea48885fc5433b828f3abb4d3bc558e1049247366041137ae10e31d5b1"} Jul 17 18:19:19.729297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:19.729256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" event={"ID":"4953ecfb-aa53-46cf-a313-481b07f060b2","Type":"ContainerStarted","Data":"f8e2de5cf8b1a63cfc89d94c17f22329bc04828c3499c3a65fba1f867323aac6"} Jul 17 18:19:19.747991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:19.747920 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" podStartSLOduration=1.498890271 podStartE2EDuration="2.747899958s" totalImagesPullingTime="1.249009687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:19:19.745846474 +0000 UTC m=+570.160584527" watchObservedRunningTime="2026-07-17 18:19:19.747899958 +0000 UTC m=+570.162637990" Jul 17 18:19:27.813912 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:27.813871 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:27.814401 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:27.813948 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:27.826488 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:27.826452 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:28.789781 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:28.789753 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9" Jul 17 18:19:49.986657 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:49.986628 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:19:49.994191 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:19:49.994169 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:21:13.463811 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.463771 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86bd5d5fbc-5tzqt"] Jul 17 18:21:13.466243 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.466223 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.475094 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.475069 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86bd5d5fbc-5tzqt"] Jul 17 18:21:13.483101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.483077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c22be84f-0691-46e6-a7d1-851775186304-tls-cert\") pod \"authorino-86bd5d5fbc-5tzqt\" (UID: \"c22be84f-0691-46e6-a7d1-851775186304\") " pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.483245 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.483131 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdmx\" (UniqueName: \"kubernetes.io/projected/c22be84f-0691-46e6-a7d1-851775186304-kube-api-access-wtdmx\") pod \"authorino-86bd5d5fbc-5tzqt\" (UID: \"c22be84f-0691-46e6-a7d1-851775186304\") " pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.584374 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.584337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c22be84f-0691-46e6-a7d1-851775186304-tls-cert\") pod \"authorino-86bd5d5fbc-5tzqt\" (UID: \"c22be84f-0691-46e6-a7d1-851775186304\") " pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.584539 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.584398 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdmx\" (UniqueName: \"kubernetes.io/projected/c22be84f-0691-46e6-a7d1-851775186304-kube-api-access-wtdmx\") pod \"authorino-86bd5d5fbc-5tzqt\" (UID: \"c22be84f-0691-46e6-a7d1-851775186304\") " pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.586808 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.586789 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c22be84f-0691-46e6-a7d1-851775186304-tls-cert\") pod \"authorino-86bd5d5fbc-5tzqt\" (UID: \"c22be84f-0691-46e6-a7d1-851775186304\") " pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.592542 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.592520 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdmx\" (UniqueName: \"kubernetes.io/projected/c22be84f-0691-46e6-a7d1-851775186304-kube-api-access-wtdmx\") pod \"authorino-86bd5d5fbc-5tzqt\" (UID: \"c22be84f-0691-46e6-a7d1-851775186304\") " pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.776118 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.775990 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" Jul 17 18:21:13.915113 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.915077 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86bd5d5fbc-5tzqt"] Jul 17 18:21:13.916659 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:21:13.916632 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc22be84f_0691_46e6_a7d1_851775186304.slice/crio-f473d8467ef448784e2b522bb768f8d554d4faf5510af9b8534ab3b161a46dac WatchSource:0}: Error finding container f473d8467ef448784e2b522bb768f8d554d4faf5510af9b8534ab3b161a46dac: Status 404 returned error can't find the container with id f473d8467ef448784e2b522bb768f8d554d4faf5510af9b8534ab3b161a46dac Jul 17 18:21:13.918427 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:13.918410 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:21:14.342489 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:14.342460 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" event={"ID":"c22be84f-0691-46e6-a7d1-851775186304","Type":"ContainerStarted","Data":"f473d8467ef448784e2b522bb768f8d554d4faf5510af9b8534ab3b161a46dac"} Jul 17 18:21:15.352147 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.352105 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" event={"ID":"c22be84f-0691-46e6-a7d1-851775186304","Type":"ContainerStarted","Data":"0d889de98226460a230ec2a51c87d803a3c22a5af9b50f1aad892b94df39ef9e"} Jul 17 18:21:15.369454 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.369385 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-86bd5d5fbc-5tzqt" podStartSLOduration=1.954174971 podStartE2EDuration="2.369344671s" totalImagesPullingTime="415.1697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:21:15.368534112 +0000 UTC m=+685.783272152" watchObservedRunningTime="2026-07-17 18:21:15.369344671 +0000 UTC m=+685.784082700" Jul 17 18:21:15.402872 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.402841 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ff649c79d-njsgv"] Jul 17 18:21:15.403098 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.403072 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7ff649c79d-njsgv" podUID="91b11c87-68ae-4976-9f1b-2ec738ce1b37" containerName="authorino" containerID="cri-o://b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6" gracePeriod=30 Jul 17 18:21:15.677693 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.677672 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:21:15.702868 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.702828 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91b11c87-68ae-4976-9f1b-2ec738ce1b37-tls-cert\") pod \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " Jul 17 18:21:15.703048 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.702883 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mspmh\" (UniqueName: \"kubernetes.io/projected/91b11c87-68ae-4976-9f1b-2ec738ce1b37-kube-api-access-mspmh\") pod \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\" (UID: \"91b11c87-68ae-4976-9f1b-2ec738ce1b37\") " Jul 17 18:21:15.705023 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.704994 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b11c87-68ae-4976-9f1b-2ec738ce1b37-kube-api-access-mspmh" (OuterVolumeSpecName: "kube-api-access-mspmh") pod "91b11c87-68ae-4976-9f1b-2ec738ce1b37" (UID: "91b11c87-68ae-4976-9f1b-2ec738ce1b37"). InnerVolumeSpecName "kube-api-access-mspmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:21:15.714367 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.714330 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b11c87-68ae-4976-9f1b-2ec738ce1b37-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "91b11c87-68ae-4976-9f1b-2ec738ce1b37" (UID: "91b11c87-68ae-4976-9f1b-2ec738ce1b37"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:21:15.804023 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.803946 2570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91b11c87-68ae-4976-9f1b-2ec738ce1b37-tls-cert\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:21:15.804023 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:15.804022 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mspmh\" (UniqueName: \"kubernetes.io/projected/91b11c87-68ae-4976-9f1b-2ec738ce1b37-kube-api-access-mspmh\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:21:16.359162 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.359072 2570 generic.go:356] "Generic (PLEG): container finished" podID="91b11c87-68ae-4976-9f1b-2ec738ce1b37" containerID="b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6" exitCode=0 Jul 17 18:21:16.359162 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.359129 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff649c79d-njsgv" Jul 17 18:21:16.359662 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.359156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff649c79d-njsgv" event={"ID":"91b11c87-68ae-4976-9f1b-2ec738ce1b37","Type":"ContainerDied","Data":"b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6"} Jul 17 18:21:16.359662 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.359208 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff649c79d-njsgv" event={"ID":"91b11c87-68ae-4976-9f1b-2ec738ce1b37","Type":"ContainerDied","Data":"3cd54c0499edf2737bbbbd81ab98a6b3eefffeb3ac36883f4f9849a9b7aa3fa5"} Jul 17 18:21:16.359662 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.359249 2570 scope.go:117] "RemoveContainer" containerID="b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6" Jul 17 18:21:16.370306 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.370280 2570 scope.go:117] "RemoveContainer" containerID="b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6" Jul 17 18:21:16.370665 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:21:16.370642 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6\": container with ID starting with b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6 not found: ID does not exist" containerID="b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6" Jul 17 18:21:16.370752 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.370674 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6"} err="failed to get container status \"b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6\": rpc error: code = NotFound desc = could not find container \"b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6\": container with ID starting with b4323ffe779b04ee026f25ef71d155c9fd9517d0f4d7cb709386f3a6c4d615c6 not found: ID does not exist" Jul 17 18:21:16.378345 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.378315 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ff649c79d-njsgv"] Jul 17 18:21:16.384792 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:16.384764 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7ff649c79d-njsgv"] Jul 17 18:21:18.073256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:21:18.073224 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91b11c87-68ae-4976-9f1b-2ec738ce1b37" path="/var/lib/kubelet/pods/91b11c87-68ae-4976-9f1b-2ec738ce1b37/volumes" Jul 17 18:24:50.025729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:24:50.025657 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:24:50.029987 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:24:50.029964 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:28:30.200690 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:30.200627 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc"] Jul 17 18:28:30.203172 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:30.200890 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" podUID="588d7f5c-a88c-4039-8ba1-1b6f13536835" containerName="manager" containerID="cri-o://aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69" gracePeriod=10 Jul 17 18:28:33.055661 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.055636 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:28:33.141872 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.141787 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588d7f5c-a88c-4039-8ba1-1b6f13536835-extensions-socket-volume\") pod \"588d7f5c-a88c-4039-8ba1-1b6f13536835\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " Jul 17 18:28:33.142064 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.141897 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkt8p\" (UniqueName: \"kubernetes.io/projected/588d7f5c-a88c-4039-8ba1-1b6f13536835-kube-api-access-qkt8p\") pod \"588d7f5c-a88c-4039-8ba1-1b6f13536835\" (UID: \"588d7f5c-a88c-4039-8ba1-1b6f13536835\") " Jul 17 18:28:33.142198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.142173 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588d7f5c-a88c-4039-8ba1-1b6f13536835-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "588d7f5c-a88c-4039-8ba1-1b6f13536835" (UID: "588d7f5c-a88c-4039-8ba1-1b6f13536835"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:28:33.143908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.143882 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588d7f5c-a88c-4039-8ba1-1b6f13536835-kube-api-access-qkt8p" (OuterVolumeSpecName: "kube-api-access-qkt8p") pod "588d7f5c-a88c-4039-8ba1-1b6f13536835" (UID: "588d7f5c-a88c-4039-8ba1-1b6f13536835"). InnerVolumeSpecName "kube-api-access-qkt8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:28:33.242448 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.242410 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkt8p\" (UniqueName: \"kubernetes.io/projected/588d7f5c-a88c-4039-8ba1-1b6f13536835-kube-api-access-qkt8p\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:28:33.242448 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.242440 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588d7f5c-a88c-4039-8ba1-1b6f13536835-extensions-socket-volume\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:28:33.646623 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.646587 2570 generic.go:356] "Generic (PLEG): container finished" podID="588d7f5c-a88c-4039-8ba1-1b6f13536835" containerID="aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69" exitCode=0 Jul 17 18:28:33.646830 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.646633 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" Jul 17 18:28:33.646830 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.646672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" event={"ID":"588d7f5c-a88c-4039-8ba1-1b6f13536835","Type":"ContainerDied","Data":"aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69"} Jul 17 18:28:33.646830 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.646728 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc" event={"ID":"588d7f5c-a88c-4039-8ba1-1b6f13536835","Type":"ContainerDied","Data":"455d3e201eeae3fc8c25c014d9122d20cd8ea96ce3d0425cea069be374643ff0"} Jul 17 18:28:33.646830 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.646750 2570 scope.go:117] "RemoveContainer" containerID="aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69" Jul 17 18:28:33.663105 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.663084 2570 scope.go:117] "RemoveContainer" containerID="aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69" Jul 17 18:28:33.663379 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:28:33.663358 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69\": container with ID starting with aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69 not found: ID does not exist" containerID="aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69" Jul 17 18:28:33.663422 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.663389 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69"} err="failed to get container status \"aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69\": rpc error: code = NotFound desc = could not find container \"aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69\": container with ID starting with aa62077312e4e43ca19fc27a2499efed7e8c2d0945c696fde59cc8ea591f9c69 not found: ID does not exist" Jul 17 18:28:33.675198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.675168 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc"] Jul 17 18:28:33.684484 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:33.684459 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wsbpc"] Jul 17 18:28:34.072164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:28:34.072128 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588d7f5c-a88c-4039-8ba1-1b6f13536835" path="/var/lib/kubelet/pods/588d7f5c-a88c-4039-8ba1-1b6f13536835/volumes" Jul 17 18:29:31.362729 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.362694 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw"] Jul 17 18:29:31.363236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.363045 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b11c87-68ae-4976-9f1b-2ec738ce1b37" containerName="authorino" Jul 17 18:29:31.363236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.363058 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b11c87-68ae-4976-9f1b-2ec738ce1b37" containerName="authorino" Jul 17 18:29:31.363236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.363069 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588d7f5c-a88c-4039-8ba1-1b6f13536835" containerName="manager" Jul 17 18:29:31.363236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.363075 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="588d7f5c-a88c-4039-8ba1-1b6f13536835" containerName="manager" Jul 17 18:29:31.363236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.363144 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b11c87-68ae-4976-9f1b-2ec738ce1b37" containerName="authorino" Jul 17 18:29:31.363236 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.363154 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="588d7f5c-a88c-4039-8ba1-1b6f13536835" containerName="manager" Jul 17 18:29:31.366084 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.366064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.368609 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.368578 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hhgcn\"" Jul 17 18:29:31.378904 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.378878 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw"] Jul 17 18:29:31.439817 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.439775 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc1e0fff-6947-4365-878e-282091ed545b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-krrgw\" (UID: \"cc1e0fff-6947-4365-878e-282091ed545b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.440014 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.439856 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw7p\" (UniqueName: \"kubernetes.io/projected/cc1e0fff-6947-4365-878e-282091ed545b-kube-api-access-fdw7p\") pod \"kuadrant-operator-controller-manager-86878c769f-krrgw\" (UID: \"cc1e0fff-6947-4365-878e-282091ed545b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.540411 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.540373 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw7p\" (UniqueName: \"kubernetes.io/projected/cc1e0fff-6947-4365-878e-282091ed545b-kube-api-access-fdw7p\") pod \"kuadrant-operator-controller-manager-86878c769f-krrgw\" (UID: \"cc1e0fff-6947-4365-878e-282091ed545b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.540587 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.540433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc1e0fff-6947-4365-878e-282091ed545b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-krrgw\" (UID: \"cc1e0fff-6947-4365-878e-282091ed545b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.540766 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.540747 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc1e0fff-6947-4365-878e-282091ed545b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-krrgw\" (UID: \"cc1e0fff-6947-4365-878e-282091ed545b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.550848 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.550824 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw7p\" (UniqueName: \"kubernetes.io/projected/cc1e0fff-6947-4365-878e-282091ed545b-kube-api-access-fdw7p\") pod \"kuadrant-operator-controller-manager-86878c769f-krrgw\" (UID: \"cc1e0fff-6947-4365-878e-282091ed545b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.677554 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.677455 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.824892 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.824842 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw"] Jul 17 18:29:31.830785 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:29:31.830747 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1e0fff_6947_4365_878e_282091ed545b.slice/crio-fc548a6795cfc51382d2ffde433c72002b472ee643375fba1f2d16481f4de114 WatchSource:0}: Error finding container fc548a6795cfc51382d2ffde433c72002b472ee643375fba1f2d16481f4de114: Status 404 returned error can't find the container with id fc548a6795cfc51382d2ffde433c72002b472ee643375fba1f2d16481f4de114 Jul 17 18:29:31.950235 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.950198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" event={"ID":"cc1e0fff-6947-4365-878e-282091ed545b","Type":"ContainerStarted","Data":"22add186d038596cba7e57d340516524a2c3161815cf0e6f81f0e745d1b10a67"} Jul 17 18:29:31.950448 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.950257 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" event={"ID":"cc1e0fff-6947-4365-878e-282091ed545b","Type":"ContainerStarted","Data":"fc548a6795cfc51382d2ffde433c72002b472ee643375fba1f2d16481f4de114"} Jul 17 18:29:31.950448 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.950312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:31.972866 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:31.972810 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" podStartSLOduration=0.972794048 podStartE2EDuration="972.794048ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:29:31.971552675 +0000 UTC m=+1182.386290706" watchObservedRunningTime="2026-07-17 18:29:31.972794048 +0000 UTC m=+1182.387532075" Jul 17 18:29:42.959766 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:42.959731 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-krrgw" Jul 17 18:29:50.060566 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:50.060541 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:29:50.061394 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:29:50.061375 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:30:00.728790 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:00.728749 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9"] Jul 17 18:30:00.732325 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:00.732303 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9" Jul 17 18:30:00.735320 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:00.735297 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:30:00.741810 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:00.741767 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9"] Jul 17 18:30:00.743240 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:00.743222 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9" Jul 17 18:30:00.885133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:00.885105 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9"] Jul 17 18:30:00.886768 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:30:00.886738 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7f416d_4f3f_40ec_8455_52b68d6b6773.slice/crio-2b041265888c942d1d7efc3ce2444298ea5af42a5c06c1d73890aee5866629b4 WatchSource:0}: Error finding container 2b041265888c942d1d7efc3ce2444298ea5af42a5c06c1d73890aee5866629b4: Status 404 returned error can't find the container with id 2b041265888c942d1d7efc3ce2444298ea5af42a5c06c1d73890aee5866629b4 Jul 17 18:30:00.888637 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:00.888620 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:30:01.105706 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:01.105615 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9" event={"ID":"9d7f416d-4f3f-40ec-8455-52b68d6b6773","Type":"ContainerStarted","Data":"2b041265888c942d1d7efc3ce2444298ea5af42a5c06c1d73890aee5866629b4"} Jul 17 18:30:05.129794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:05.129751 2570 generic.go:356] "Generic (PLEG): container finished" podID="9d7f416d-4f3f-40ec-8455-52b68d6b6773" containerID="7036ae882a78f058c36df8bec53bda90f05d4ed59552940f5ab4fbc41c6baebc" exitCode=0 Jul 17 18:30:05.130294 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:05.129837 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9" event={"ID":"9d7f416d-4f3f-40ec-8455-52b68d6b6773","Type":"ContainerDied","Data":"7036ae882a78f058c36df8bec53bda90f05d4ed59552940f5ab4fbc41c6baebc"} Jul 17 18:30:06.268104 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:06.268078 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9" Jul 17 18:30:07.141841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:07.141803 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9" event={"ID":"9d7f416d-4f3f-40ec-8455-52b68d6b6773","Type":"ContainerDied","Data":"2b041265888c942d1d7efc3ce2444298ea5af42a5c06c1d73890aee5866629b4"} Jul 17 18:30:07.141841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:07.141826 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-c67w9" Jul 17 18:30:07.142099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:30:07.141857 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b041265888c942d1d7efc3ce2444298ea5af42a5c06c1d73890aee5866629b4" Jul 17 18:34:50.097485 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:34:50.097451 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:34:50.100065 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:34:50.097527 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:36:18.434675 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.434641 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb"] Jul 17 18:36:18.435145 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.434998 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d7f416d-4f3f-40ec-8455-52b68d6b6773" containerName="cleanup" Jul 17 18:36:18.435145 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.435009 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d7f416d-4f3f-40ec-8455-52b68d6b6773" containerName="cleanup" Jul 17 18:36:18.435145 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.435076 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d7f416d-4f3f-40ec-8455-52b68d6b6773" containerName="cleanup" Jul 17 18:36:18.438177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.438158 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" Jul 17 18:36:18.441082 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.441061 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:36:18.447342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.447319 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb"] Jul 17 18:36:18.515669 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.515623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb\" (UID: \"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" Jul 17 18:36:18.617036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.617005 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb\" (UID: \"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" Jul 17 18:36:18.617701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.617678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb\" (UID: \"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" Jul 17 18:36:18.748994 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.748877 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" Jul 17 18:36:18.887964 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:18.887924 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb"] Jul 17 18:36:18.889175 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:36:18.889131 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c48d1cc_5380_4d4f_944d_7ea6f7e63c9d.slice/crio-a77874861aaca524e3e5d636e6e462313798901f54e91c45c4eff3ebe260e67d WatchSource:0}: Error finding container a77874861aaca524e3e5d636e6e462313798901f54e91c45c4eff3ebe260e67d: Status 404 returned error can't find the container with id a77874861aaca524e3e5d636e6e462313798901f54e91c45c4eff3ebe260e67d Jul 17 18:36:19.156920 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:19.156876 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" event={"ID":"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d","Type":"ContainerStarted","Data":"a77874861aaca524e3e5d636e6e462313798901f54e91c45c4eff3ebe260e67d"} Jul 17 18:36:20.163994 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:20.163930 2570 generic.go:356] "Generic (PLEG): container finished" podID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerID="30efc9f18a5a9dcccc903fb421bea259fd6b026d1a7b3c5d039ed51e3ab7ac1a" exitCode=7 Jul 17 18:36:20.164431 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:20.163995 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" event={"ID":"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d","Type":"ContainerDied","Data":"30efc9f18a5a9dcccc903fb421bea259fd6b026d1a7b3c5d039ed51e3ab7ac1a"} Jul 17 18:36:20.164431 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:20.164250 2570 scope.go:117] "RemoveContainer" containerID="30efc9f18a5a9dcccc903fb421bea259fd6b026d1a7b3c5d039ed51e3ab7ac1a" Jul 17 18:36:21.170751 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:21.170659 2570 generic.go:356] "Generic (PLEG): container finished" podID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerID="60fcecd3afdf4a3274c474479511900823cbea3581627693062238c346572f35" exitCode=7 Jul 17 18:36:21.170751 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:21.170739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" event={"ID":"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d","Type":"ContainerDied","Data":"60fcecd3afdf4a3274c474479511900823cbea3581627693062238c346572f35"} Jul 17 18:36:21.171248 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:21.170795 2570 scope.go:117] "RemoveContainer" containerID="30efc9f18a5a9dcccc903fb421bea259fd6b026d1a7b3c5d039ed51e3ab7ac1a" Jul 17 18:36:21.171248 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:21.171000 2570 scope.go:117] "RemoveContainer" containerID="60fcecd3afdf4a3274c474479511900823cbea3581627693062238c346572f35" Jul 17 18:36:21.171248 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:36:21.171233 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb_odh-ai-gateway-infra(8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" Jul 17 18:36:22.178366 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:22.178340 2570 scope.go:117] "RemoveContainer" containerID="60fcecd3afdf4a3274c474479511900823cbea3581627693062238c346572f35" Jul 17 18:36:22.178744 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:36:22.178509 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb_odh-ai-gateway-infra(8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" Jul 17 18:36:36.067340 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:36.067306 2570 scope.go:117] "RemoveContainer" containerID="60fcecd3afdf4a3274c474479511900823cbea3581627693062238c346572f35" Jul 17 18:36:37.261068 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:37.261033 2570 generic.go:356] "Generic (PLEG): container finished" podID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerID="974ba2a876d9f4130adeddcf5b9ade99a235ecf9c6de8e55fd530f50355950c3" exitCode=0 Jul 17 18:36:37.261482 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:37.261082 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" event={"ID":"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d","Type":"ContainerDied","Data":"974ba2a876d9f4130adeddcf5b9ade99a235ecf9c6de8e55fd530f50355950c3"} Jul 17 18:36:37.261482 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:37.261132 2570 scope.go:117] "RemoveContainer" containerID="60fcecd3afdf4a3274c474479511900823cbea3581627693062238c346572f35" Jul 17 18:36:38.293473 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:38.293439 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb"] Jul 17 18:36:38.399768 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:38.399741 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" Jul 17 18:36:38.496783 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:38.496751 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d-maas-api-service-ca\") pod \"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d\" (UID: \"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d\") " Jul 17 18:36:38.497180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:38.497158 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" (UID: "8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:36:38.597501 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:38.597409 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:36:39.274832 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:39.274797 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" event={"ID":"8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d","Type":"ContainerDied","Data":"a77874861aaca524e3e5d636e6e462313798901f54e91c45c4eff3ebe260e67d"} Jul 17 18:36:39.274832 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:39.274822 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb" Jul 17 18:36:39.275117 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:39.274845 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a77874861aaca524e3e5d636e6e462313798901f54e91c45c4eff3ebe260e67d" Jul 17 18:36:39.306939 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:39.306905 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb"] Jul 17 18:36:39.312001 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:39.311968 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-r45rb"] Jul 17 18:36:40.072478 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:36:40.072443 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" path="/var/lib/kubelet/pods/8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d/volumes" Jul 17 18:37:18.488107 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488021 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f"] Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488359 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488370 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488390 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488395 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488402 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488408 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488457 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.488565 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.488466 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:18.491680 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.491659 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" Jul 17 18:37:18.494238 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.494218 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:37:18.499268 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.499235 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f"] Jul 17 18:37:18.618455 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.618418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4442376-3267-46fa-aeca-4289d36167db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f\" (UID: \"e4442376-3267-46fa-aeca-4289d36167db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" Jul 17 18:37:18.719796 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.719753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4442376-3267-46fa-aeca-4289d36167db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f\" (UID: \"e4442376-3267-46fa-aeca-4289d36167db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" Jul 17 18:37:18.720418 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.720398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4442376-3267-46fa-aeca-4289d36167db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f\" (UID: \"e4442376-3267-46fa-aeca-4289d36167db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" Jul 17 18:37:18.803409 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.803328 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" Jul 17 18:37:18.945840 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:18.945815 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f"] Jul 17 18:37:18.948310 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:37:18.948274 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4442376_3267_46fa_aeca_4289d36167db.slice/crio-66887041a902de9ca0f22fd1facfedb94aeab785994a2547e64467b7f79f6410 WatchSource:0}: Error finding container 66887041a902de9ca0f22fd1facfedb94aeab785994a2547e64467b7f79f6410: Status 404 returned error can't find the container with id 66887041a902de9ca0f22fd1facfedb94aeab785994a2547e64467b7f79f6410 Jul 17 18:37:19.479746 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:19.479712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" event={"ID":"e4442376-3267-46fa-aeca-4289d36167db","Type":"ContainerStarted","Data":"b942cc80a294bf6ec8052f05a827465d5f9011409838b3d954bcb72d1305b03e"} Jul 17 18:37:19.479946 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:19.479773 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" event={"ID":"e4442376-3267-46fa-aeca-4289d36167db","Type":"ContainerStarted","Data":"66887041a902de9ca0f22fd1facfedb94aeab785994a2547e64467b7f79f6410"} Jul 17 18:37:19.497185 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:19.497135 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" podStartSLOduration=1.4971186570000001 podStartE2EDuration="1.497118657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:19.494775108 +0000 UTC m=+1649.909513134" watchObservedRunningTime="2026-07-17 18:37:19.497118657 +0000 UTC m=+1649.911856683" Jul 17 18:37:20.486633 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:20.486590 2570 generic.go:356] "Generic (PLEG): container finished" podID="e4442376-3267-46fa-aeca-4289d36167db" containerID="b942cc80a294bf6ec8052f05a827465d5f9011409838b3d954bcb72d1305b03e" exitCode=0 Jul 17 18:37:20.486633 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:20.486632 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" event={"ID":"e4442376-3267-46fa-aeca-4289d36167db","Type":"ContainerDied","Data":"b942cc80a294bf6ec8052f05a827465d5f9011409838b3d954bcb72d1305b03e"} Jul 17 18:37:21.626132 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:21.626108 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" Jul 17 18:37:21.746823 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:21.746740 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4442376-3267-46fa-aeca-4289d36167db-maas-api-service-ca\") pod \"e4442376-3267-46fa-aeca-4289d36167db\" (UID: \"e4442376-3267-46fa-aeca-4289d36167db\") " Jul 17 18:37:21.747165 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:21.747138 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4442376-3267-46fa-aeca-4289d36167db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4442376-3267-46fa-aeca-4289d36167db" (UID: "e4442376-3267-46fa-aeca-4289d36167db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:37:21.848357 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:21.848323 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4442376-3267-46fa-aeca-4289d36167db-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:37:22.499031 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:22.498994 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" event={"ID":"e4442376-3267-46fa-aeca-4289d36167db","Type":"ContainerDied","Data":"66887041a902de9ca0f22fd1facfedb94aeab785994a2547e64467b7f79f6410"} Jul 17 18:37:22.499213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:22.499049 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66887041a902de9ca0f22fd1facfedb94aeab785994a2547e64467b7f79f6410" Jul 17 18:37:22.499213 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:22.499016 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f" Jul 17 18:37:45.993485 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:45.993451 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk"] Jul 17 18:37:45.994056 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:45.993972 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4442376-3267-46fa-aeca-4289d36167db" containerName="revoke-keys" Jul 17 18:37:45.994056 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:45.993987 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4442376-3267-46fa-aeca-4289d36167db" containerName="revoke-keys" Jul 17 18:37:45.994193 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:45.994076 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4442376-3267-46fa-aeca-4289d36167db" containerName="revoke-keys" Jul 17 18:37:45.994193 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:45.994091 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c48d1cc-5380-4d4f-944d-7ea6f7e63c9d" containerName="revoke-keys" Jul 17 18:37:45.997559 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:45.997536 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.000275 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.000252 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d37232e3-serving-cert\"" Jul 17 18:37:46.000275 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.000268 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:37:46.010884 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.010860 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk"] Jul 17 18:37:46.050651 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.050618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-openshift-service-ca\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.050815 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.050681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-maas-api-tls\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.050815 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.050718 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqcsp\" (UniqueName: \"kubernetes.io/projected/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-kube-api-access-dqcsp\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.151457 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.151416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-maas-api-tls\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.151642 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.151473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqcsp\" (UniqueName: \"kubernetes.io/projected/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-kube-api-access-dqcsp\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.151642 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.151498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-openshift-service-ca\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.152194 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.152168 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-openshift-service-ca\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.154009 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.153985 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-maas-api-tls\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.160529 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.160508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqcsp\" (UniqueName: \"kubernetes.io/projected/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-kube-api-access-dqcsp\") pod \"maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.310700 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.310613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:46.450520 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.450490 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk"] Jul 17 18:37:46.452156 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:37:46.452126 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9b1e14_20bc_42eb_aaf0_29cca88b1ae6.slice/crio-92f08945104f859c9d4f8bae4656fbafc5eff2f8b1d6d25ea7565dec9808013e WatchSource:0}: Error finding container 92f08945104f859c9d4f8bae4656fbafc5eff2f8b1d6d25ea7565dec9808013e: Status 404 returned error can't find the container with id 92f08945104f859c9d4f8bae4656fbafc5eff2f8b1d6d25ea7565dec9808013e Jul 17 18:37:46.453829 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.453812 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:37:46.631484 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:46.631391 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" event={"ID":"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6","Type":"ContainerStarted","Data":"92f08945104f859c9d4f8bae4656fbafc5eff2f8b1d6d25ea7565dec9808013e"} Jul 17 18:37:49.650900 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:49.650864 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" event={"ID":"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6","Type":"ContainerStarted","Data":"42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4"} Jul 17 18:37:49.651375 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:49.651002 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:37:49.669935 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:49.669885 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" podStartSLOduration=2.341592947 podStartE2EDuration="4.669870392s" totalImagesPullingTime="2.328277445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:49.667918674 +0000 UTC m=+1680.082656703" watchObservedRunningTime="2026-07-17 18:37:49.669870392 +0000 UTC m=+1680.084608418" Jul 17 18:37:50.214657 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.214625 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc"] Jul 17 18:37:50.217917 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.217895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" Jul 17 18:37:50.220622 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.220601 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:37:50.228785 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.228760 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc"] Jul 17 18:37:50.288525 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.288492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36060395-efc7-438b-91d3-128c1fdcd3a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc\" (UID: \"36060395-efc7-438b-91d3-128c1fdcd3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" Jul 17 18:37:50.389681 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.389640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36060395-efc7-438b-91d3-128c1fdcd3a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc\" (UID: \"36060395-efc7-438b-91d3-128c1fdcd3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" Jul 17 18:37:50.390355 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.390336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36060395-efc7-438b-91d3-128c1fdcd3a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc\" (UID: \"36060395-efc7-438b-91d3-128c1fdcd3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" Jul 17 18:37:50.530879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.530788 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" Jul 17 18:37:50.678337 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:50.678306 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc"] Jul 17 18:37:50.681698 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:37:50.681669 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36060395_efc7_438b_91d3_128c1fdcd3a3.slice/crio-17b188a3c8896da1f02f93cd470eaf31d808e091528be10ff8ae131086879e1a WatchSource:0}: Error finding container 17b188a3c8896da1f02f93cd470eaf31d808e091528be10ff8ae131086879e1a: Status 404 returned error can't find the container with id 17b188a3c8896da1f02f93cd470eaf31d808e091528be10ff8ae131086879e1a Jul 17 18:37:51.671158 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:51.671063 2570 generic.go:356] "Generic (PLEG): container finished" podID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerID="a00bf2eb7209ac2ee00e64c44cba198d324b50b7ac746da6aa898553cf350141" exitCode=7 Jul 17 18:37:51.671158 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:51.671148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" event={"ID":"36060395-efc7-438b-91d3-128c1fdcd3a3","Type":"ContainerDied","Data":"a00bf2eb7209ac2ee00e64c44cba198d324b50b7ac746da6aa898553cf350141"} Jul 17 18:37:51.671388 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:51.671213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" event={"ID":"36060395-efc7-438b-91d3-128c1fdcd3a3","Type":"ContainerStarted","Data":"17b188a3c8896da1f02f93cd470eaf31d808e091528be10ff8ae131086879e1a"} Jul 17 18:37:51.671388 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:51.671340 2570 scope.go:117] "RemoveContainer" containerID="a00bf2eb7209ac2ee00e64c44cba198d324b50b7ac746da6aa898553cf350141" Jul 17 18:37:52.678971 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:52.678912 2570 generic.go:356] "Generic (PLEG): container finished" podID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerID="48c9b1115d58be068966354976d309e678e21ecd73512e03befb641b7c05fc7a" exitCode=7 Jul 17 18:37:52.679363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:52.679004 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" event={"ID":"36060395-efc7-438b-91d3-128c1fdcd3a3","Type":"ContainerDied","Data":"48c9b1115d58be068966354976d309e678e21ecd73512e03befb641b7c05fc7a"} Jul 17 18:37:52.679363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:52.679066 2570 scope.go:117] "RemoveContainer" containerID="a00bf2eb7209ac2ee00e64c44cba198d324b50b7ac746da6aa898553cf350141" Jul 17 18:37:52.679363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:52.679253 2570 scope.go:117] "RemoveContainer" containerID="48c9b1115d58be068966354976d309e678e21ecd73512e03befb641b7c05fc7a" Jul 17 18:37:52.679502 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:37:52.679485 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc_odh-ai-gateway-infra(36060395-efc7-438b-91d3-128c1fdcd3a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" Jul 17 18:37:53.686545 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:53.686519 2570 scope.go:117] "RemoveContainer" containerID="48c9b1115d58be068966354976d309e678e21ecd73512e03befb641b7c05fc7a" Jul 17 18:37:53.686919 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:37:53.686716 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc_odh-ai-gateway-infra(36060395-efc7-438b-91d3-128c1fdcd3a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" Jul 17 18:37:55.663833 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:37:55.663807 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:38:09.066689 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:09.066648 2570 scope.go:117] "RemoveContainer" containerID="48c9b1115d58be068966354976d309e678e21ecd73512e03befb641b7c05fc7a" Jul 17 18:38:09.791323 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:09.791292 2570 generic.go:356] "Generic (PLEG): container finished" podID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerID="b67e8636ad01d4b817e06f52b109c3ad776fae6187a09403aeded9bb64ea03fa" exitCode=0 Jul 17 18:38:09.791491 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:09.791363 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" event={"ID":"36060395-efc7-438b-91d3-128c1fdcd3a3","Type":"ContainerDied","Data":"b67e8636ad01d4b817e06f52b109c3ad776fae6187a09403aeded9bb64ea03fa"} Jul 17 18:38:09.791491 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:09.791427 2570 scope.go:117] "RemoveContainer" containerID="48c9b1115d58be068966354976d309e678e21ecd73512e03befb641b7c05fc7a" Jul 17 18:38:10.823412 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:10.823379 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc"] Jul 17 18:38:10.934224 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:10.934197 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" Jul 17 18:38:11.058327 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.058233 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36060395-efc7-438b-91d3-128c1fdcd3a3-maas-api-service-ca\") pod \"36060395-efc7-438b-91d3-128c1fdcd3a3\" (UID: \"36060395-efc7-438b-91d3-128c1fdcd3a3\") " Jul 17 18:38:11.058601 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.058577 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36060395-efc7-438b-91d3-128c1fdcd3a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36060395-efc7-438b-91d3-128c1fdcd3a3" (UID: "36060395-efc7-438b-91d3-128c1fdcd3a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:38:11.159115 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.159078 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36060395-efc7-438b-91d3-128c1fdcd3a3-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:38:11.804491 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.804460 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" Jul 17 18:38:11.804642 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.804491 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc" event={"ID":"36060395-efc7-438b-91d3-128c1fdcd3a3","Type":"ContainerDied","Data":"17b188a3c8896da1f02f93cd470eaf31d808e091528be10ff8ae131086879e1a"} Jul 17 18:38:11.804642 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.804538 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b188a3c8896da1f02f93cd470eaf31d808e091528be10ff8ae131086879e1a" Jul 17 18:38:11.835805 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.835770 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc"] Jul 17 18:38:11.839405 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:11.839379 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-f9bcc"] Jul 17 18:38:12.073791 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:12.073697 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" path="/var/lib/kubelet/pods/36060395-efc7-438b-91d3-128c1fdcd3a3/volumes" Jul 17 18:38:50.261773 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.261734 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm"] Jul 17 18:38:50.262250 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.262090 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:38:50.262250 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.262101 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:38:50.262250 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.262119 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:38:50.262250 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.262125 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:38:50.262250 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.262179 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:38:50.262250 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.262187 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:38:50.267072 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.267052 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" Jul 17 18:38:50.270194 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.270168 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:38:50.272990 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.272930 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm"] Jul 17 18:38:50.288354 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.288325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54926b45-223b-43ae-85a3-99a1ff06e4d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm\" (UID: \"54926b45-223b-43ae-85a3-99a1ff06e4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" Jul 17 18:38:50.389343 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.389303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54926b45-223b-43ae-85a3-99a1ff06e4d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm\" (UID: \"54926b45-223b-43ae-85a3-99a1ff06e4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" Jul 17 18:38:50.389969 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.389926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54926b45-223b-43ae-85a3-99a1ff06e4d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm\" (UID: \"54926b45-223b-43ae-85a3-99a1ff06e4d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" Jul 17 18:38:50.580126 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.580036 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" Jul 17 18:38:50.716899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:50.716865 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm"] Jul 17 18:38:50.718699 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:38:50.718671 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54926b45_223b_43ae_85a3_99a1ff06e4d0.slice/crio-7947ef9f99a030e03c5f78d9ff288ce281a557484d88d910579494b5eb5e9032 WatchSource:0}: Error finding container 7947ef9f99a030e03c5f78d9ff288ce281a557484d88d910579494b5eb5e9032: Status 404 returned error can't find the container with id 7947ef9f99a030e03c5f78d9ff288ce281a557484d88d910579494b5eb5e9032 Jul 17 18:38:51.011120 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:51.011091 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" event={"ID":"54926b45-223b-43ae-85a3-99a1ff06e4d0","Type":"ContainerStarted","Data":"7947ef9f99a030e03c5f78d9ff288ce281a557484d88d910579494b5eb5e9032"} Jul 17 18:38:52.018348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:52.018314 2570 generic.go:356] "Generic (PLEG): container finished" podID="54926b45-223b-43ae-85a3-99a1ff06e4d0" containerID="8fd3bb4872d97e48d7ec9d5f177cc484775351c8271cadfec6f91e5a43a51a3c" exitCode=0 Jul 17 18:38:52.018843 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:52.018417 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" event={"ID":"54926b45-223b-43ae-85a3-99a1ff06e4d0","Type":"ContainerDied","Data":"8fd3bb4872d97e48d7ec9d5f177cc484775351c8271cadfec6f91e5a43a51a3c"} Jul 17 18:38:53.156685 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:53.156660 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" Jul 17 18:38:53.211457 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:53.211425 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54926b45-223b-43ae-85a3-99a1ff06e4d0-maas-api-service-ca\") pod \"54926b45-223b-43ae-85a3-99a1ff06e4d0\" (UID: \"54926b45-223b-43ae-85a3-99a1ff06e4d0\") " Jul 17 18:38:53.211757 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:53.211736 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54926b45-223b-43ae-85a3-99a1ff06e4d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54926b45-223b-43ae-85a3-99a1ff06e4d0" (UID: "54926b45-223b-43ae-85a3-99a1ff06e4d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:38:53.312563 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:53.312475 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54926b45-223b-43ae-85a3-99a1ff06e4d0-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:38:54.031210 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:54.031171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" event={"ID":"54926b45-223b-43ae-85a3-99a1ff06e4d0","Type":"ContainerDied","Data":"7947ef9f99a030e03c5f78d9ff288ce281a557484d88d910579494b5eb5e9032"} Jul 17 18:38:54.031210 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:54.031199 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm" Jul 17 18:38:54.031435 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:54.031234 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7947ef9f99a030e03c5f78d9ff288ce281a557484d88d910579494b5eb5e9032" Jul 17 18:38:56.084097 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.084065 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk"] Jul 17 18:38:56.084571 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.084302 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" podUID="0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" containerName="maas-api" containerID="cri-o://42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4" gracePeriod=30 Jul 17 18:38:56.341267 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.341206 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:38:56.433864 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.433832 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-openshift-service-ca\") pod \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " Jul 17 18:38:56.433864 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.433875 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqcsp\" (UniqueName: \"kubernetes.io/projected/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-kube-api-access-dqcsp\") pod \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " Jul 17 18:38:56.434109 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.433918 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-maas-api-tls\") pod \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\" (UID: \"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6\") " Jul 17 18:38:56.434238 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.434214 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" (UID: "0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:38:56.436160 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.436136 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" (UID: "0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:38:56.436277 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.436179 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-kube-api-access-dqcsp" (OuterVolumeSpecName: "kube-api-access-dqcsp") pod "0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" (UID: "0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6"). InnerVolumeSpecName "kube-api-access-dqcsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:38:56.535092 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.535056 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:38:56.535092 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.535087 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:38:56.535092 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:56.535099 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqcsp\" (UniqueName: \"kubernetes.io/projected/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6-kube-api-access-dqcsp\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:38:57.048132 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.048095 2570 generic.go:356] "Generic (PLEG): container finished" podID="0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" containerID="42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4" exitCode=0 Jul 17 18:38:57.048302 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.048138 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" Jul 17 18:38:57.048302 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.048151 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" event={"ID":"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6","Type":"ContainerDied","Data":"42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4"} Jul 17 18:38:57.048302 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.048198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk" event={"ID":"0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6","Type":"ContainerDied","Data":"92f08945104f859c9d4f8bae4656fbafc5eff2f8b1d6d25ea7565dec9808013e"} Jul 17 18:38:57.048302 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.048223 2570 scope.go:117] "RemoveContainer" containerID="42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4" Jul 17 18:38:57.057879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.057854 2570 scope.go:117] "RemoveContainer" containerID="42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4" Jul 17 18:38:57.058194 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:38:57.058175 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4\": container with ID starting with 42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4 not found: ID does not exist" containerID="42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4" Jul 17 18:38:57.058266 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.058205 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4"} err="failed to get container status \"42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4\": rpc error: code = NotFound desc = could not find container \"42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4\": container with ID starting with 42046c13cc8c9a7b49722235c795da64c644c34f7f12de308c89afe2f75349f4 not found: ID does not exist" Jul 17 18:38:57.070041 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.070015 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk"] Jul 17 18:38:57.075130 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:57.075106 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d37232e3-6d8d6568d8-bjhpk"] Jul 17 18:38:58.072040 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:38:58.072006 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" path="/var/lib/kubelet/pods/0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6/volumes" Jul 17 18:39:03.554190 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.554140 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5"] Jul 17 18:39:03.555232 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555205 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" containerName="maas-api" Jul 17 18:39:03.555232 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555232 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" containerName="maas-api" Jul 17 18:39:03.555436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555245 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:39:03.555436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555255 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:39:03.555436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555269 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54926b45-223b-43ae-85a3-99a1ff06e4d0" containerName="revoke-keys" Jul 17 18:39:03.555436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555277 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="54926b45-223b-43ae-85a3-99a1ff06e4d0" containerName="revoke-keys" Jul 17 18:39:03.555436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555373 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9b1e14-20bc-42eb-aaf0-29cca88b1ae6" containerName="maas-api" Jul 17 18:39:03.555436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555384 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="36060395-efc7-438b-91d3-128c1fdcd3a3" containerName="revoke-keys" Jul 17 18:39:03.555436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.555396 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="54926b45-223b-43ae-85a3-99a1ff06e4d0" containerName="revoke-keys" Jul 17 18:39:03.560074 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.560055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.562774 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.562753 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:39:03.562879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.562791 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9b0be972-serving-cert\"" Jul 17 18:39:03.574133 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.574108 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5"] Jul 17 18:39:03.601688 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.601658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7062672-f7f0-4fb6-abec-16f33cff95a4-openshift-service-ca\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.601688 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.601691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgqsn\" (UniqueName: \"kubernetes.io/projected/d7062672-f7f0-4fb6-abec-16f33cff95a4-kube-api-access-vgqsn\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.601891 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.601725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.703121 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.703082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7062672-f7f0-4fb6-abec-16f33cff95a4-openshift-service-ca\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.703121 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.703122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgqsn\" (UniqueName: \"kubernetes.io/projected/d7062672-f7f0-4fb6-abec-16f33cff95a4-kube-api-access-vgqsn\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.703330 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.703222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.703369 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:39:03.703340 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-serving-cert: secret "maas-api-e2e-ait-9b0be972-serving-cert" not found Jul 17 18:39:03.703424 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:39:03.703415 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls podName:d7062672-f7f0-4fb6-abec-16f33cff95a4 nodeName:}" failed. No retries permitted until 2026-07-17 18:39:04.203399902 +0000 UTC m=+1754.618137907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls") pod "maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" (UID: "d7062672-f7f0-4fb6-abec-16f33cff95a4") : secret "maas-api-e2e-ait-9b0be972-serving-cert" not found Jul 17 18:39:03.703746 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.703728 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7062672-f7f0-4fb6-abec-16f33cff95a4-openshift-service-ca\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:03.748448 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:03.748419 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgqsn\" (UniqueName: \"kubernetes.io/projected/d7062672-f7f0-4fb6-abec-16f33cff95a4-kube-api-access-vgqsn\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:04.208112 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:04.208074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:04.210848 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:04.210819 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls\") pod \"maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:04.473174 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:04.473082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:04.623395 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:04.623369 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5"] Jul 17 18:39:04.625736 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:39:04.625711 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7062672_f7f0_4fb6_abec_16f33cff95a4.slice/crio-96a0c7520610a408aea2ca8236f57bf3243c1bdf9adf9a0eb3b8a00918a9b679 WatchSource:0}: Error finding container 96a0c7520610a408aea2ca8236f57bf3243c1bdf9adf9a0eb3b8a00918a9b679: Status 404 returned error can't find the container with id 96a0c7520610a408aea2ca8236f57bf3243c1bdf9adf9a0eb3b8a00918a9b679 Jul 17 18:39:05.096091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:05.096058 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" event={"ID":"d7062672-f7f0-4fb6-abec-16f33cff95a4","Type":"ContainerStarted","Data":"96a0c7520610a408aea2ca8236f57bf3243c1bdf9adf9a0eb3b8a00918a9b679"} Jul 17 18:39:06.102739 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:06.102689 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" event={"ID":"d7062672-f7f0-4fb6-abec-16f33cff95a4","Type":"ContainerStarted","Data":"442b776e7894b9ea7e47a07ba4d01626786b3f49f170aea0157735c9f9479b65"} Jul 17 18:39:06.103147 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:06.102762 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:06.123355 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:06.123299 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" podStartSLOduration=2.68523976 podStartE2EDuration="3.123281827s" totalImagesPullingTime="438.042067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:06.121439014 +0000 UTC m=+1756.536177041" watchObservedRunningTime="2026-07-17 18:39:06.123281827 +0000 UTC m=+1756.538019853" Jul 17 18:39:08.575612 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.575574 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j"] Jul 17 18:39:08.579099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.579080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" Jul 17 18:39:08.582159 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.582138 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:39:08.588227 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.588205 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j"] Jul 17 18:39:08.648990 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.648934 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e34d94-742c-46a1-a040-37a3f7385103-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j\" (UID: \"00e34d94-742c-46a1-a040-37a3f7385103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" Jul 17 18:39:08.750136 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.750090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e34d94-742c-46a1-a040-37a3f7385103-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j\" (UID: \"00e34d94-742c-46a1-a040-37a3f7385103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" Jul 17 18:39:08.750732 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.750708 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e34d94-742c-46a1-a040-37a3f7385103-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j\" (UID: \"00e34d94-742c-46a1-a040-37a3f7385103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" Jul 17 18:39:08.891055 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:08.890942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" Jul 17 18:39:09.029739 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:09.029707 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j"] Jul 17 18:39:09.030880 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:39:09.030852 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e34d94_742c_46a1_a040_37a3f7385103.slice/crio-96b2f2c77627747ff99cd647410dd88fe95e4955c9d72cab85abff4436450a7f WatchSource:0}: Error finding container 96b2f2c77627747ff99cd647410dd88fe95e4955c9d72cab85abff4436450a7f: Status 404 returned error can't find the container with id 96b2f2c77627747ff99cd647410dd88fe95e4955c9d72cab85abff4436450a7f Jul 17 18:39:09.122880 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:09.122843 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" event={"ID":"00e34d94-742c-46a1-a040-37a3f7385103","Type":"ContainerStarted","Data":"96b2f2c77627747ff99cd647410dd88fe95e4955c9d72cab85abff4436450a7f"} Jul 17 18:39:10.130528 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:10.130492 2570 generic.go:356] "Generic (PLEG): container finished" podID="00e34d94-742c-46a1-a040-37a3f7385103" containerID="3a4cc6e7d280b1f9c9b5f697d1155f7e01ee7ed5f123fb3834686f20b1794f32" exitCode=7 Jul 17 18:39:10.130528 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:10.130522 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" event={"ID":"00e34d94-742c-46a1-a040-37a3f7385103","Type":"ContainerDied","Data":"3a4cc6e7d280b1f9c9b5f697d1155f7e01ee7ed5f123fb3834686f20b1794f32"} Jul 17 18:39:10.131019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:10.130774 2570 scope.go:117] "RemoveContainer" containerID="3a4cc6e7d280b1f9c9b5f697d1155f7e01ee7ed5f123fb3834686f20b1794f32" Jul 17 18:39:11.138493 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:11.138395 2570 generic.go:356] "Generic (PLEG): container finished" podID="00e34d94-742c-46a1-a040-37a3f7385103" containerID="ba0487f5def03f7a9e129781081dcf00e69b042eec04d5ae10b8946b9db17406" exitCode=7 Jul 17 18:39:11.138493 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:11.138476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" event={"ID":"00e34d94-742c-46a1-a040-37a3f7385103","Type":"ContainerDied","Data":"ba0487f5def03f7a9e129781081dcf00e69b042eec04d5ae10b8946b9db17406"} Jul 17 18:39:11.139006 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:11.138540 2570 scope.go:117] "RemoveContainer" containerID="3a4cc6e7d280b1f9c9b5f697d1155f7e01ee7ed5f123fb3834686f20b1794f32" Jul 17 18:39:11.139006 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:11.138706 2570 scope.go:117] "RemoveContainer" containerID="ba0487f5def03f7a9e129781081dcf00e69b042eec04d5ae10b8946b9db17406" Jul 17 18:39:11.139006 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:39:11.138928 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j_odh-ai-gateway-infra(00e34d94-742c-46a1-a040-37a3f7385103)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" podUID="00e34d94-742c-46a1-a040-37a3f7385103" Jul 17 18:39:12.114116 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:12.114088 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:39:12.148363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:12.148333 2570 scope.go:117] "RemoveContainer" containerID="ba0487f5def03f7a9e129781081dcf00e69b042eec04d5ae10b8946b9db17406" Jul 17 18:39:12.148893 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:39:12.148583 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j_odh-ai-gateway-infra(00e34d94-742c-46a1-a040-37a3f7385103)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" podUID="00e34d94-742c-46a1-a040-37a3f7385103" Jul 17 18:39:24.066770 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:24.066735 2570 scope.go:117] "RemoveContainer" containerID="ba0487f5def03f7a9e129781081dcf00e69b042eec04d5ae10b8946b9db17406" Jul 17 18:39:25.220284 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:25.220250 2570 generic.go:356] "Generic (PLEG): container finished" podID="00e34d94-742c-46a1-a040-37a3f7385103" containerID="e708e93b8845a4e575f73b9724ef38bc267dd91dc7935722e589aa49b9f51524" exitCode=0 Jul 17 18:39:25.220761 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:25.220305 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" event={"ID":"00e34d94-742c-46a1-a040-37a3f7385103","Type":"ContainerDied","Data":"e708e93b8845a4e575f73b9724ef38bc267dd91dc7935722e589aa49b9f51524"} Jul 17 18:39:25.220761 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:25.220354 2570 scope.go:117] "RemoveContainer" containerID="ba0487f5def03f7a9e129781081dcf00e69b042eec04d5ae10b8946b9db17406" Jul 17 18:39:26.264546 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:26.264511 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j"] Jul 17 18:39:26.361719 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:26.361695 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" Jul 17 18:39:26.504791 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:26.504702 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e34d94-742c-46a1-a040-37a3f7385103-maas-api-service-ca\") pod \"00e34d94-742c-46a1-a040-37a3f7385103\" (UID: \"00e34d94-742c-46a1-a040-37a3f7385103\") " Jul 17 18:39:26.505077 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:26.505055 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e34d94-742c-46a1-a040-37a3f7385103-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00e34d94-742c-46a1-a040-37a3f7385103" (UID: "00e34d94-742c-46a1-a040-37a3f7385103"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:39:26.606199 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:26.606157 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00e34d94-742c-46a1-a040-37a3f7385103-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:39:27.234542 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:27.234516 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" Jul 17 18:39:27.234721 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:27.234561 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j" event={"ID":"00e34d94-742c-46a1-a040-37a3f7385103","Type":"ContainerDied","Data":"96b2f2c77627747ff99cd647410dd88fe95e4955c9d72cab85abff4436450a7f"} Jul 17 18:39:27.234721 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:27.234609 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96b2f2c77627747ff99cd647410dd88fe95e4955c9d72cab85abff4436450a7f" Jul 17 18:39:27.264109 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:27.264069 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j"] Jul 17 18:39:27.274293 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:27.269475 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-hxn5j"] Jul 17 18:39:28.072588 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:28.072556 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e34d94-742c-46a1-a040-37a3f7385103" path="/var/lib/kubelet/pods/00e34d94-742c-46a1-a040-37a3f7385103/volumes" Jul 17 18:39:50.132057 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:50.131917 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:39:50.132057 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:50.131944 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:39:58.651819 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.651786 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55"] Jul 17 18:39:58.652289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.652185 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:39:58.652289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.652198 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:39:58.652289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.652211 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:39:58.652289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.652216 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:39:58.652289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.652273 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:39:58.652289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.652281 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:39:58.652289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.652287 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:39:58.655384 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.655368 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" Jul 17 18:39:58.658277 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.658259 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:39:58.675303 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.675275 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55"] Jul 17 18:39:58.678601 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.678571 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27eb318b-ea60-4146-8a3e-aeafbe944bcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b0be972-w4r55\" (UID: \"27eb318b-ea60-4146-8a3e-aeafbe944bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" Jul 17 18:39:58.779718 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.779683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27eb318b-ea60-4146-8a3e-aeafbe944bcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b0be972-w4r55\" (UID: \"27eb318b-ea60-4146-8a3e-aeafbe944bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" Jul 17 18:39:58.780351 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.780332 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27eb318b-ea60-4146-8a3e-aeafbe944bcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b0be972-w4r55\" (UID: \"27eb318b-ea60-4146-8a3e-aeafbe944bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" Jul 17 18:39:58.965908 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:58.965876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" Jul 17 18:39:59.097755 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:59.097728 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55"] Jul 17 18:39:59.099602 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:39:59.099566 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27eb318b_ea60_4146_8a3e_aeafbe944bcc.slice/crio-d68fcf21a570c2319799d3435219fff819bee41dc367a6dab927fbfb72654e24 WatchSource:0}: Error finding container d68fcf21a570c2319799d3435219fff819bee41dc367a6dab927fbfb72654e24: Status 404 returned error can't find the container with id d68fcf21a570c2319799d3435219fff819bee41dc367a6dab927fbfb72654e24 Jul 17 18:39:59.417367 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:39:59.417329 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" event={"ID":"27eb318b-ea60-4146-8a3e-aeafbe944bcc","Type":"ContainerStarted","Data":"d68fcf21a570c2319799d3435219fff819bee41dc367a6dab927fbfb72654e24"} Jul 17 18:40:00.424707 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:00.424669 2570 generic.go:356] "Generic (PLEG): container finished" podID="27eb318b-ea60-4146-8a3e-aeafbe944bcc" containerID="587fdb58d204699ff88652833fab61defa9401d54a783bcbdc443eadb7d8557a" exitCode=0 Jul 17 18:40:00.425151 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:00.424755 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" event={"ID":"27eb318b-ea60-4146-8a3e-aeafbe944bcc","Type":"ContainerDied","Data":"587fdb58d204699ff88652833fab61defa9401d54a783bcbdc443eadb7d8557a"} Jul 17 18:40:01.567767 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:01.567744 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" Jul 17 18:40:01.602832 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:01.602802 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27eb318b-ea60-4146-8a3e-aeafbe944bcc-maas-api-service-ca\") pod \"27eb318b-ea60-4146-8a3e-aeafbe944bcc\" (UID: \"27eb318b-ea60-4146-8a3e-aeafbe944bcc\") " Jul 17 18:40:01.603153 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:01.603129 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27eb318b-ea60-4146-8a3e-aeafbe944bcc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27eb318b-ea60-4146-8a3e-aeafbe944bcc" (UID: "27eb318b-ea60-4146-8a3e-aeafbe944bcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:01.704584 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:01.704485 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27eb318b-ea60-4146-8a3e-aeafbe944bcc-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:40:02.436530 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:02.436503 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" Jul 17 18:40:02.436714 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:02.436502 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b0be972-w4r55" event={"ID":"27eb318b-ea60-4146-8a3e-aeafbe944bcc","Type":"ContainerDied","Data":"d68fcf21a570c2319799d3435219fff819bee41dc367a6dab927fbfb72654e24"} Jul 17 18:40:02.436714 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:02.436626 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68fcf21a570c2319799d3435219fff819bee41dc367a6dab927fbfb72654e24" Jul 17 18:40:09.288125 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.288056 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5"] Jul 17 18:40:09.288705 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.288291 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" podUID="d7062672-f7f0-4fb6-abec-16f33cff95a4" containerName="maas-api" containerID="cri-o://442b776e7894b9ea7e47a07ba4d01626786b3f49f170aea0157735c9f9479b65" gracePeriod=30 Jul 17 18:40:09.486470 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.486436 2570 generic.go:356] "Generic (PLEG): container finished" podID="d7062672-f7f0-4fb6-abec-16f33cff95a4" containerID="442b776e7894b9ea7e47a07ba4d01626786b3f49f170aea0157735c9f9479b65" exitCode=0 Jul 17 18:40:09.486698 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.486492 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" event={"ID":"d7062672-f7f0-4fb6-abec-16f33cff95a4","Type":"ContainerDied","Data":"442b776e7894b9ea7e47a07ba4d01626786b3f49f170aea0157735c9f9479b65"} Jul 17 18:40:09.538083 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.538059 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:40:09.673927 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.673838 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7062672-f7f0-4fb6-abec-16f33cff95a4-openshift-service-ca\") pod \"d7062672-f7f0-4fb6-abec-16f33cff95a4\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " Jul 17 18:40:09.674110 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.673978 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls\") pod \"d7062672-f7f0-4fb6-abec-16f33cff95a4\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " Jul 17 18:40:09.674110 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.674011 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgqsn\" (UniqueName: \"kubernetes.io/projected/d7062672-f7f0-4fb6-abec-16f33cff95a4-kube-api-access-vgqsn\") pod \"d7062672-f7f0-4fb6-abec-16f33cff95a4\" (UID: \"d7062672-f7f0-4fb6-abec-16f33cff95a4\") " Jul 17 18:40:09.674275 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.674249 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7062672-f7f0-4fb6-abec-16f33cff95a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d7062672-f7f0-4fb6-abec-16f33cff95a4" (UID: "d7062672-f7f0-4fb6-abec-16f33cff95a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:09.676129 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.676098 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7062672-f7f0-4fb6-abec-16f33cff95a4-kube-api-access-vgqsn" (OuterVolumeSpecName: "kube-api-access-vgqsn") pod "d7062672-f7f0-4fb6-abec-16f33cff95a4" (UID: "d7062672-f7f0-4fb6-abec-16f33cff95a4"). InnerVolumeSpecName "kube-api-access-vgqsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:40:09.676239 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.676144 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d7062672-f7f0-4fb6-abec-16f33cff95a4" (UID: "d7062672-f7f0-4fb6-abec-16f33cff95a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:40:09.774826 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.774785 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7062672-f7f0-4fb6-abec-16f33cff95a4-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:40:09.774826 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.774820 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7062672-f7f0-4fb6-abec-16f33cff95a4-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:40:09.774826 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:09.774829 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgqsn\" (UniqueName: \"kubernetes.io/projected/d7062672-f7f0-4fb6-abec-16f33cff95a4-kube-api-access-vgqsn\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:40:10.493257 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:10.493228 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" Jul 17 18:40:10.493257 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:10.493246 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5" event={"ID":"d7062672-f7f0-4fb6-abec-16f33cff95a4","Type":"ContainerDied","Data":"96a0c7520610a408aea2ca8236f57bf3243c1bdf9adf9a0eb3b8a00918a9b679"} Jul 17 18:40:10.493751 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:10.493306 2570 scope.go:117] "RemoveContainer" containerID="442b776e7894b9ea7e47a07ba4d01626786b3f49f170aea0157735c9f9479b65" Jul 17 18:40:10.515397 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:10.515373 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5"] Jul 17 18:40:10.522452 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:10.522430 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b0be972-69866ccff6-dzcx5"] Jul 17 18:40:12.073065 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:12.073031 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7062672-f7f0-4fb6-abec-16f33cff95a4" path="/var/lib/kubelet/pods/d7062672-f7f0-4fb6-abec-16f33cff95a4/volumes" Jul 17 18:40:26.451073 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451037 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv"] Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451385 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27eb318b-ea60-4146-8a3e-aeafbe944bcc" containerName="revoke-keys" Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451401 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="27eb318b-ea60-4146-8a3e-aeafbe944bcc" containerName="revoke-keys" Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451420 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7062672-f7f0-4fb6-abec-16f33cff95a4" containerName="maas-api" Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451429 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7062672-f7f0-4fb6-abec-16f33cff95a4" containerName="maas-api" Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451445 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451460 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e34d94-742c-46a1-a040-37a3f7385103" containerName="revoke-keys" Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451540 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7062672-f7f0-4fb6-abec-16f33cff95a4" containerName="maas-api" Jul 17 18:40:26.451562 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.451559 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="27eb318b-ea60-4146-8a3e-aeafbe944bcc" containerName="revoke-keys" Jul 17 18:40:26.454849 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.454825 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.458070 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.458043 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1b52ad55-serving-cert\"" Jul 17 18:40:26.458070 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.458065 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:40:26.468439 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.468417 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv"] Jul 17 18:40:26.517008 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.516944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mnf\" (UniqueName: \"kubernetes.io/projected/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-kube-api-access-68mnf\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.517187 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.517027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.517187 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.517113 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-openshift-service-ca\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.618018 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.617973 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-openshift-service-ca\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.618177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.618053 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mnf\" (UniqueName: \"kubernetes.io/projected/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-kube-api-access-68mnf\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.618177 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.618092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.618285 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:40:26.618222 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-serving-cert: secret "maas-api-e2e-derive-1b52ad55-serving-cert" not found Jul 17 18:40:26.618342 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:40:26.618286 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls podName:d0a5495d-ed8c-4659-96eb-cb61ec80cc2e nodeName:}" failed. No retries permitted until 2026-07-17 18:40:27.118266877 +0000 UTC m=+1837.533004884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls") pod "maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" (UID: "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e") : secret "maas-api-e2e-derive-1b52ad55-serving-cert" not found Jul 17 18:40:26.618813 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.618789 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-openshift-service-ca\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:26.627925 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:26.627896 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mnf\" (UniqueName: \"kubernetes.io/projected/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-kube-api-access-68mnf\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:27.122878 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:27.122837 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:27.123077 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:40:27.123018 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-serving-cert: secret "maas-api-e2e-derive-1b52ad55-serving-cert" not found Jul 17 18:40:27.123137 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:40:27.123096 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls podName:d0a5495d-ed8c-4659-96eb-cb61ec80cc2e nodeName:}" failed. No retries permitted until 2026-07-17 18:40:28.123079792 +0000 UTC m=+1838.537817800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls") pod "maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" (UID: "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e") : secret "maas-api-e2e-derive-1b52ad55-serving-cert" not found Jul 17 18:40:28.132037 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:28.131993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:28.134465 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:28.134436 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls\") pod \"maas-api-e2e-derive-1b52ad55-f998784c6-n78qv\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:28.265711 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:28.265676 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:28.401271 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:28.401239 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv"] Jul 17 18:40:28.402562 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:40:28.402533 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a5495d_ed8c_4659_96eb_cb61ec80cc2e.slice/crio-6e3d905fa951a1fef43a43b22be3ff7a47f5411064be81dc7f7e47c99a73d12b WatchSource:0}: Error finding container 6e3d905fa951a1fef43a43b22be3ff7a47f5411064be81dc7f7e47c99a73d12b: Status 404 returned error can't find the container with id 6e3d905fa951a1fef43a43b22be3ff7a47f5411064be81dc7f7e47c99a73d12b Jul 17 18:40:28.592340 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:28.592307 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" event={"ID":"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e","Type":"ContainerStarted","Data":"6e3d905fa951a1fef43a43b22be3ff7a47f5411064be81dc7f7e47c99a73d12b"} Jul 17 18:40:29.599484 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:29.599438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" event={"ID":"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e","Type":"ContainerStarted","Data":"bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712"} Jul 17 18:40:29.599891 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:29.599521 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:29.617832 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:29.617774 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" podStartSLOduration=3.2567685539999998 podStartE2EDuration="3.617753075s" totalImagesPullingTime="360.984521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:40:29.616852909 +0000 UTC m=+1840.031590937" watchObservedRunningTime="2026-07-17 18:40:29.617753075 +0000 UTC m=+1840.032491102" Jul 17 18:40:30.584737 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.584701 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267"] Jul 17 18:40:30.588064 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.588042 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" Jul 17 18:40:30.591016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.590994 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:40:30.598296 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.598270 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267"] Jul 17 18:40:30.756114 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.756072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/360b7261-ec82-4806-9878-0faab19bc13c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b52ad55-5s267\" (UID: \"360b7261-ec82-4806-9878-0faab19bc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" Jul 17 18:40:30.856985 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.856878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/360b7261-ec82-4806-9878-0faab19bc13c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b52ad55-5s267\" (UID: \"360b7261-ec82-4806-9878-0faab19bc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" Jul 17 18:40:30.857539 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.857518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/360b7261-ec82-4806-9878-0faab19bc13c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b52ad55-5s267\" (UID: \"360b7261-ec82-4806-9878-0faab19bc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" Jul 17 18:40:30.899761 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:30.899726 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" Jul 17 18:40:31.031063 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:31.031037 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267"] Jul 17 18:40:31.032548 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:40:31.032518 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360b7261_ec82_4806_9878_0faab19bc13c.slice/crio-476fbdd9eb87676b89c1444ca3684417c43fff8cbc8e7eeb80cdc36dab1f59c2 WatchSource:0}: Error finding container 476fbdd9eb87676b89c1444ca3684417c43fff8cbc8e7eeb80cdc36dab1f59c2: Status 404 returned error can't find the container with id 476fbdd9eb87676b89c1444ca3684417c43fff8cbc8e7eeb80cdc36dab1f59c2 Jul 17 18:40:31.612649 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:31.612603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" event={"ID":"360b7261-ec82-4806-9878-0faab19bc13c","Type":"ContainerStarted","Data":"92a691c77ec4ab64464f65197ca699065fb85a2055bcd1f63590ed7c5496b3a2"} Jul 17 18:40:31.612865 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:31.612666 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" event={"ID":"360b7261-ec82-4806-9878-0faab19bc13c","Type":"ContainerStarted","Data":"476fbdd9eb87676b89c1444ca3684417c43fff8cbc8e7eeb80cdc36dab1f59c2"} Jul 17 18:40:31.630531 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:31.630482 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" podStartSLOduration=1.6304642440000001 podStartE2EDuration="1.630464244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:40:31.628696417 +0000 UTC m=+1842.043434449" watchObservedRunningTime="2026-07-17 18:40:31.630464244 +0000 UTC m=+1842.045202270" Jul 17 18:40:32.620422 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:32.620387 2570 generic.go:356] "Generic (PLEG): container finished" podID="360b7261-ec82-4806-9878-0faab19bc13c" containerID="92a691c77ec4ab64464f65197ca699065fb85a2055bcd1f63590ed7c5496b3a2" exitCode=7 Jul 17 18:40:32.620839 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:32.620428 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" event={"ID":"360b7261-ec82-4806-9878-0faab19bc13c","Type":"ContainerDied","Data":"92a691c77ec4ab64464f65197ca699065fb85a2055bcd1f63590ed7c5496b3a2"} Jul 17 18:40:32.620839 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:32.620660 2570 scope.go:117] "RemoveContainer" containerID="92a691c77ec4ab64464f65197ca699065fb85a2055bcd1f63590ed7c5496b3a2" Jul 17 18:40:33.628012 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:33.627909 2570 generic.go:356] "Generic (PLEG): container finished" podID="360b7261-ec82-4806-9878-0faab19bc13c" containerID="98297ca97dae76cdcc355bd2427bef26761d5127b1741c1937bb0615213d6ee0" exitCode=7 Jul 17 18:40:33.628012 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:33.627993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" event={"ID":"360b7261-ec82-4806-9878-0faab19bc13c","Type":"ContainerDied","Data":"98297ca97dae76cdcc355bd2427bef26761d5127b1741c1937bb0615213d6ee0"} Jul 17 18:40:33.628492 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:33.628052 2570 scope.go:117] "RemoveContainer" containerID="92a691c77ec4ab64464f65197ca699065fb85a2055bcd1f63590ed7c5496b3a2" Jul 17 18:40:33.628492 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:33.628187 2570 scope.go:117] "RemoveContainer" containerID="98297ca97dae76cdcc355bd2427bef26761d5127b1741c1937bb0615213d6ee0" Jul 17 18:40:33.628492 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:40:33.628410 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1b52ad55-5s267_odh-ai-gateway-infra(360b7261-ec82-4806-9878-0faab19bc13c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" podUID="360b7261-ec82-4806-9878-0faab19bc13c" Jul 17 18:40:34.635293 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:34.635267 2570 scope.go:117] "RemoveContainer" containerID="98297ca97dae76cdcc355bd2427bef26761d5127b1741c1937bb0615213d6ee0" Jul 17 18:40:34.635681 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:40:34.635478 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1b52ad55-5s267_odh-ai-gateway-infra(360b7261-ec82-4806-9878-0faab19bc13c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" podUID="360b7261-ec82-4806-9878-0faab19bc13c" Jul 17 18:40:35.611674 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:35.611645 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:40:49.067003 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:49.066941 2570 scope.go:117] "RemoveContainer" containerID="98297ca97dae76cdcc355bd2427bef26761d5127b1741c1937bb0615213d6ee0" Jul 17 18:40:49.719932 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:49.719899 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" event={"ID":"360b7261-ec82-4806-9878-0faab19bc13c","Type":"ContainerStarted","Data":"61c23d787c6eb714836359e80113ed3bebc31f3983e4de2a255796ebbc90479c"} Jul 17 18:40:50.726574 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:50.726542 2570 generic.go:356] "Generic (PLEG): container finished" podID="360b7261-ec82-4806-9878-0faab19bc13c" containerID="61c23d787c6eb714836359e80113ed3bebc31f3983e4de2a255796ebbc90479c" exitCode=0 Jul 17 18:40:50.727028 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:50.726622 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" event={"ID":"360b7261-ec82-4806-9878-0faab19bc13c","Type":"ContainerDied","Data":"61c23d787c6eb714836359e80113ed3bebc31f3983e4de2a255796ebbc90479c"} Jul 17 18:40:50.727028 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:50.726690 2570 scope.go:117] "RemoveContainer" containerID="98297ca97dae76cdcc355bd2427bef26761d5127b1741c1937bb0615213d6ee0" Jul 17 18:40:50.751196 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:50.751162 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267"] Jul 17 18:40:51.874373 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:51.874345 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" Jul 17 18:40:51.930253 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:51.930224 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/360b7261-ec82-4806-9878-0faab19bc13c-maas-api-service-ca\") pod \"360b7261-ec82-4806-9878-0faab19bc13c\" (UID: \"360b7261-ec82-4806-9878-0faab19bc13c\") " Jul 17 18:40:51.930574 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:51.930552 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360b7261-ec82-4806-9878-0faab19bc13c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "360b7261-ec82-4806-9878-0faab19bc13c" (UID: "360b7261-ec82-4806-9878-0faab19bc13c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:52.030791 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:52.030702 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/360b7261-ec82-4806-9878-0faab19bc13c-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:40:52.747909 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:52.747879 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" Jul 17 18:40:52.747909 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:52.747886 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267" event={"ID":"360b7261-ec82-4806-9878-0faab19bc13c","Type":"ContainerDied","Data":"476fbdd9eb87676b89c1444ca3684417c43fff8cbc8e7eeb80cdc36dab1f59c2"} Jul 17 18:40:52.748182 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:52.747935 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="476fbdd9eb87676b89c1444ca3684417c43fff8cbc8e7eeb80cdc36dab1f59c2" Jul 17 18:40:52.779202 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:52.779170 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267"] Jul 17 18:40:52.783386 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:52.783362 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-5s267"] Jul 17 18:40:54.072761 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:40:54.072718 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="360b7261-ec82-4806-9878-0faab19bc13c" path="/var/lib/kubelet/pods/360b7261-ec82-4806-9878-0faab19bc13c/volumes" Jul 17 18:41:30.638382 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.638344 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q"] Jul 17 18:41:30.638827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.638689 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:41:30.638827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.638700 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:41:30.638827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.638708 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:41:30.638827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.638713 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:41:30.638827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.638775 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:41:30.638827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.638784 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:41:30.641858 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.641841 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" Jul 17 18:41:30.645195 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.645172 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:41:30.650528 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.650495 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q"] Jul 17 18:41:30.760890 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.760849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727d16f5-baf2-4f1c-a4c4-5792fe8cec92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q\" (UID: \"727d16f5-baf2-4f1c-a4c4-5792fe8cec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" Jul 17 18:41:30.861937 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.861898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727d16f5-baf2-4f1c-a4c4-5792fe8cec92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q\" (UID: \"727d16f5-baf2-4f1c-a4c4-5792fe8cec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" Jul 17 18:41:30.862514 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.862493 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727d16f5-baf2-4f1c-a4c4-5792fe8cec92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q\" (UID: \"727d16f5-baf2-4f1c-a4c4-5792fe8cec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" Jul 17 18:41:30.953744 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:30.953696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" Jul 17 18:41:31.088424 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:31.088398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q"] Jul 17 18:41:31.089986 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:41:31.089941 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727d16f5_baf2_4f1c_a4c4_5792fe8cec92.slice/crio-709889d2e7b9c844b28f9131405e14fd72cda7f498730d83a9d43ade4ff15685 WatchSource:0}: Error finding container 709889d2e7b9c844b28f9131405e14fd72cda7f498730d83a9d43ade4ff15685: Status 404 returned error can't find the container with id 709889d2e7b9c844b28f9131405e14fd72cda7f498730d83a9d43ade4ff15685 Jul 17 18:41:31.978123 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:31.978091 2570 generic.go:356] "Generic (PLEG): container finished" podID="727d16f5-baf2-4f1c-a4c4-5792fe8cec92" containerID="673e5df8d0635c773be995356ddb36de1002bfb701a3946706bc27e2b4be4406" exitCode=0 Jul 17 18:41:31.978497 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:31.978135 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" event={"ID":"727d16f5-baf2-4f1c-a4c4-5792fe8cec92","Type":"ContainerDied","Data":"673e5df8d0635c773be995356ddb36de1002bfb701a3946706bc27e2b4be4406"} Jul 17 18:41:31.978497 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:31.978179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" event={"ID":"727d16f5-baf2-4f1c-a4c4-5792fe8cec92","Type":"ContainerStarted","Data":"709889d2e7b9c844b28f9131405e14fd72cda7f498730d83a9d43ade4ff15685"} Jul 17 18:41:33.121721 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:33.121693 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" Jul 17 18:41:33.281772 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:33.281689 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727d16f5-baf2-4f1c-a4c4-5792fe8cec92-maas-api-service-ca\") pod \"727d16f5-baf2-4f1c-a4c4-5792fe8cec92\" (UID: \"727d16f5-baf2-4f1c-a4c4-5792fe8cec92\") " Jul 17 18:41:33.282063 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:33.282041 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727d16f5-baf2-4f1c-a4c4-5792fe8cec92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "727d16f5-baf2-4f1c-a4c4-5792fe8cec92" (UID: "727d16f5-baf2-4f1c-a4c4-5792fe8cec92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:41:33.383059 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:33.383025 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727d16f5-baf2-4f1c-a4c4-5792fe8cec92-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:41:33.989915 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:33.989889 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" Jul 17 18:41:33.990112 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:33.989931 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q" event={"ID":"727d16f5-baf2-4f1c-a4c4-5792fe8cec92","Type":"ContainerDied","Data":"709889d2e7b9c844b28f9131405e14fd72cda7f498730d83a9d43ade4ff15685"} Jul 17 18:41:33.990112 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:33.989992 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="709889d2e7b9c844b28f9131405e14fd72cda7f498730d83a9d43ade4ff15685" Jul 17 18:41:41.084917 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.084879 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv"] Jul 17 18:41:41.085428 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.085174 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" podUID="d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" containerName="maas-api" containerID="cri-o://bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712" gracePeriod=30 Jul 17 18:41:41.337613 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.337554 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:41:41.458055 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.458017 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68mnf\" (UniqueName: \"kubernetes.io/projected/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-kube-api-access-68mnf\") pod \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " Jul 17 18:41:41.458055 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.458053 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls\") pod \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " Jul 17 18:41:41.458294 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.458162 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-openshift-service-ca\") pod \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\" (UID: \"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e\") " Jul 17 18:41:41.458520 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.458495 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" (UID: "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:41:41.460176 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.460155 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" (UID: "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:41:41.460259 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.460191 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-kube-api-access-68mnf" (OuterVolumeSpecName: "kube-api-access-68mnf") pod "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" (UID: "d0a5495d-ed8c-4659-96eb-cb61ec80cc2e"). InnerVolumeSpecName "kube-api-access-68mnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:41:41.559701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.559668 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:41:41.559701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.559695 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mnf\" (UniqueName: \"kubernetes.io/projected/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-kube-api-access-68mnf\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:41:41.559701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:41.559705 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:41:42.034940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.034902 2570 generic.go:356] "Generic (PLEG): container finished" podID="d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" containerID="bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712" exitCode=0 Jul 17 18:41:42.034940 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.034943 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" event={"ID":"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e","Type":"ContainerDied","Data":"bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712"} Jul 17 18:41:42.035243 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.035022 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" event={"ID":"d0a5495d-ed8c-4659-96eb-cb61ec80cc2e","Type":"ContainerDied","Data":"6e3d905fa951a1fef43a43b22be3ff7a47f5411064be81dc7f7e47c99a73d12b"} Jul 17 18:41:42.035243 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.034948 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv" Jul 17 18:41:42.035243 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.035050 2570 scope.go:117] "RemoveContainer" containerID="bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712" Jul 17 18:41:42.044784 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.044767 2570 scope.go:117] "RemoveContainer" containerID="bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712" Jul 17 18:41:42.045075 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:41:42.045055 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712\": container with ID starting with bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712 not found: ID does not exist" containerID="bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712" Jul 17 18:41:42.045180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.045080 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712"} err="failed to get container status \"bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712\": rpc error: code = NotFound desc = could not find container \"bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712\": container with ID starting with bef57359ab824fe5224867f623384619790c25908f0432d169679ba518389712 not found: ID does not exist" Jul 17 18:41:42.056620 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.056594 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv"] Jul 17 18:41:42.058741 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.058712 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b52ad55-f998784c6-n78qv"] Jul 17 18:41:42.077877 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:41:42.077843 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" path="/var/lib/kubelet/pods/d0a5495d-ed8c-4659-96eb-cb61ec80cc2e/volumes" Jul 17 18:42:50.231927 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:42:50.231844 2570 scope.go:117] "RemoveContainer" containerID="974ba2a876d9f4130adeddcf5b9ade99a235ecf9c6de8e55fd530f50355950c3" Jul 17 18:43:38.258995 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.258963 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk"] Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259325 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727d16f5-baf2-4f1c-a4c4-5792fe8cec92" containerName="revoke-keys" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259336 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="727d16f5-baf2-4f1c-a4c4-5792fe8cec92" containerName="revoke-keys" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259346 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" containerName="maas-api" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259351 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" containerName="maas-api" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259369 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259374 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259432 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a5495d-ed8c-4659-96eb-cb61ec80cc2e" containerName="maas-api" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259448 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="360b7261-ec82-4806-9878-0faab19bc13c" containerName="revoke-keys" Jul 17 18:43:38.259577 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.259456 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="727d16f5-baf2-4f1c-a4c4-5792fe8cec92" containerName="revoke-keys" Jul 17 18:43:38.262502 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.262484 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.265468 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.265445 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:43:38.265599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.265445 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ba643e57-serving-cert\"" Jul 17 18:43:38.274798 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.274729 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk"] Jul 17 18:43:38.359343 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.359307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvsq\" (UniqueName: \"kubernetes.io/projected/940d91e8-72db-4331-8459-525e320afde5-kube-api-access-hgvsq\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.359554 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.359382 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.359554 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.359482 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940d91e8-72db-4331-8459-525e320afde5-openshift-service-ca\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.460886 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.460847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940d91e8-72db-4331-8459-525e320afde5-openshift-service-ca\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.461107 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.460900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvsq\" (UniqueName: \"kubernetes.io/projected/940d91e8-72db-4331-8459-525e320afde5-kube-api-access-hgvsq\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.461107 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.460950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.461107 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:43:38.461054 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-serving-cert: secret "maas-api-e2e-mt-ba643e57-serving-cert" not found Jul 17 18:43:38.461296 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:43:38.461114 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls podName:940d91e8-72db-4331-8459-525e320afde5 nodeName:}" failed. No retries permitted until 2026-07-17 18:43:38.961099537 +0000 UTC m=+2029.375837542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls") pod "maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" (UID: "940d91e8-72db-4331-8459-525e320afde5") : secret "maas-api-e2e-mt-ba643e57-serving-cert" not found Jul 17 18:43:38.461650 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.461627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940d91e8-72db-4331-8459-525e320afde5-openshift-service-ca\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.470752 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.470726 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvsq\" (UniqueName: \"kubernetes.io/projected/940d91e8-72db-4331-8459-525e320afde5-kube-api-access-hgvsq\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.964864 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.964828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:38.967356 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:38.967332 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls\") pod \"maas-api-e2e-mt-ba643e57-5988cb888b-clfwk\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:39.175354 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:39.175322 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:39.314068 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:39.314038 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk"] Jul 17 18:43:39.315052 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:43:39.315021 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940d91e8_72db_4331_8459_525e320afde5.slice/crio-72f8ed784d847af02620a59b1b7187676bfb8e8f61137460211fc4169952e2a9 WatchSource:0}: Error finding container 72f8ed784d847af02620a59b1b7187676bfb8e8f61137460211fc4169952e2a9: Status 404 returned error can't find the container with id 72f8ed784d847af02620a59b1b7187676bfb8e8f61137460211fc4169952e2a9 Jul 17 18:43:39.317491 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:39.317460 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:43:39.706118 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:39.706084 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" event={"ID":"940d91e8-72db-4331-8459-525e320afde5","Type":"ContainerStarted","Data":"72f8ed784d847af02620a59b1b7187676bfb8e8f61137460211fc4169952e2a9"} Jul 17 18:43:40.713994 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:40.713940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" event={"ID":"940d91e8-72db-4331-8459-525e320afde5","Type":"ContainerStarted","Data":"1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8"} Jul 17 18:43:40.714411 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:40.714027 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:40.733245 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:40.733194 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" podStartSLOduration=2.266506703 podStartE2EDuration="2.733179159s" totalImagesPullingTime="466.672456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:40.73216422 +0000 UTC m=+2031.146902249" watchObservedRunningTime="2026-07-17 18:43:40.733179159 +0000 UTC m=+2031.147917206" Jul 17 18:43:46.727387 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:46.727359 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:43:49.835911 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:49.835877 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs"] Jul 17 18:43:49.839793 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:49.839773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" Jul 17 18:43:49.842397 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:49.842376 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:43:49.850066 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:49.850040 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs"] Jul 17 18:43:49.857142 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:49.857110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/741ebdeb-9476-45d9-81f7-91e499c1462c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs\" (UID: \"741ebdeb-9476-45d9-81f7-91e499c1462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" Jul 17 18:43:49.957696 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:49.957669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/741ebdeb-9476-45d9-81f7-91e499c1462c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs\" (UID: \"741ebdeb-9476-45d9-81f7-91e499c1462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" Jul 17 18:43:49.958324 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:49.958307 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/741ebdeb-9476-45d9-81f7-91e499c1462c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs\" (UID: \"741ebdeb-9476-45d9-81f7-91e499c1462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" Jul 17 18:43:50.154702 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:50.154624 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:43:50.163017 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:50.162996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" Jul 17 18:43:50.302848 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:50.301676 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs"] Jul 17 18:43:50.775201 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:50.775114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" event={"ID":"741ebdeb-9476-45d9-81f7-91e499c1462c","Type":"ContainerStarted","Data":"157473e70d51836c1f60550b2ca5914270479f187fda3e9ea87b3cbc12b37e5e"} Jul 17 18:43:50.775201 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:50.775166 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" event={"ID":"741ebdeb-9476-45d9-81f7-91e499c1462c","Type":"ContainerStarted","Data":"a8446d36c4b92d3c76b725352bde0a1ce755bec4b4d2f4b9ac5ac5e3f7aff220"} Jul 17 18:43:50.791818 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:50.791760 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" podStartSLOduration=1.791744922 podStartE2EDuration="1.791744922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:50.789623571 +0000 UTC m=+2041.204361612" watchObservedRunningTime="2026-07-17 18:43:50.791744922 +0000 UTC m=+2041.206482949" Jul 17 18:43:51.781745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:51.781711 2570 generic.go:356] "Generic (PLEG): container finished" podID="741ebdeb-9476-45d9-81f7-91e499c1462c" containerID="157473e70d51836c1f60550b2ca5914270479f187fda3e9ea87b3cbc12b37e5e" exitCode=0 Jul 17 18:43:51.782182 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:51.781800 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" event={"ID":"741ebdeb-9476-45d9-81f7-91e499c1462c","Type":"ContainerDied","Data":"157473e70d51836c1f60550b2ca5914270479f187fda3e9ea87b3cbc12b37e5e"} Jul 17 18:43:52.926378 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:52.926355 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" Jul 17 18:43:52.983796 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:52.983761 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/741ebdeb-9476-45d9-81f7-91e499c1462c-maas-api-service-ca\") pod \"741ebdeb-9476-45d9-81f7-91e499c1462c\" (UID: \"741ebdeb-9476-45d9-81f7-91e499c1462c\") " Jul 17 18:43:52.984183 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:52.984155 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741ebdeb-9476-45d9-81f7-91e499c1462c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "741ebdeb-9476-45d9-81f7-91e499c1462c" (UID: "741ebdeb-9476-45d9-81f7-91e499c1462c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:43:53.084661 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:53.084568 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/741ebdeb-9476-45d9-81f7-91e499c1462c-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:43:53.795081 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:53.795044 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" event={"ID":"741ebdeb-9476-45d9-81f7-91e499c1462c","Type":"ContainerDied","Data":"a8446d36c4b92d3c76b725352bde0a1ce755bec4b4d2f4b9ac5ac5e3f7aff220"} Jul 17 18:43:53.795257 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:53.795096 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8446d36c4b92d3c76b725352bde0a1ce755bec4b4d2f4b9ac5ac5e3f7aff220" Jul 17 18:43:53.795257 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:43:53.795050 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs" Jul 17 18:44:00.200198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.200100 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk"] Jul 17 18:44:00.200686 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.200346 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" podUID="940d91e8-72db-4331-8459-525e320afde5" containerName="maas-api" containerID="cri-o://1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8" gracePeriod=30 Jul 17 18:44:00.450858 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.450794 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:44:00.547009 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.546934 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls\") pod \"940d91e8-72db-4331-8459-525e320afde5\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " Jul 17 18:44:00.547186 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.547083 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940d91e8-72db-4331-8459-525e320afde5-openshift-service-ca\") pod \"940d91e8-72db-4331-8459-525e320afde5\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " Jul 17 18:44:00.547186 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.547133 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgvsq\" (UniqueName: \"kubernetes.io/projected/940d91e8-72db-4331-8459-525e320afde5-kube-api-access-hgvsq\") pod \"940d91e8-72db-4331-8459-525e320afde5\" (UID: \"940d91e8-72db-4331-8459-525e320afde5\") " Jul 17 18:44:00.547505 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.547469 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940d91e8-72db-4331-8459-525e320afde5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "940d91e8-72db-4331-8459-525e320afde5" (UID: "940d91e8-72db-4331-8459-525e320afde5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:44:00.549143 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.549123 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940d91e8-72db-4331-8459-525e320afde5-kube-api-access-hgvsq" (OuterVolumeSpecName: "kube-api-access-hgvsq") pod "940d91e8-72db-4331-8459-525e320afde5" (UID: "940d91e8-72db-4331-8459-525e320afde5"). InnerVolumeSpecName "kube-api-access-hgvsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:44:00.549224 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.549179 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "940d91e8-72db-4331-8459-525e320afde5" (UID: "940d91e8-72db-4331-8459-525e320afde5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:44:00.648488 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.648448 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940d91e8-72db-4331-8459-525e320afde5-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:44:00.648488 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.648482 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940d91e8-72db-4331-8459-525e320afde5-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:44:00.648488 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.648491 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgvsq\" (UniqueName: \"kubernetes.io/projected/940d91e8-72db-4331-8459-525e320afde5-kube-api-access-hgvsq\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:44:00.836451 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.836366 2570 generic.go:356] "Generic (PLEG): container finished" podID="940d91e8-72db-4331-8459-525e320afde5" containerID="1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8" exitCode=0 Jul 17 18:44:00.836451 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.836414 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" Jul 17 18:44:00.836644 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.836453 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" event={"ID":"940d91e8-72db-4331-8459-525e320afde5","Type":"ContainerDied","Data":"1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8"} Jul 17 18:44:00.836644 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.836509 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk" event={"ID":"940d91e8-72db-4331-8459-525e320afde5","Type":"ContainerDied","Data":"72f8ed784d847af02620a59b1b7187676bfb8e8f61137460211fc4169952e2a9"} Jul 17 18:44:00.836644 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.836538 2570 scope.go:117] "RemoveContainer" containerID="1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8" Jul 17 18:44:00.845919 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.845900 2570 scope.go:117] "RemoveContainer" containerID="1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8" Jul 17 18:44:00.846219 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:44:00.846194 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8\": container with ID starting with 1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8 not found: ID does not exist" containerID="1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8" Jul 17 18:44:00.846318 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.846224 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8"} err="failed to get container status \"1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8\": rpc error: code = NotFound desc = could not find container \"1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8\": container with ID starting with 1d0ae1737d250c8a12a91b293290e3ab36a3e8ce3fe89efad520f48584f530f8 not found: ID does not exist" Jul 17 18:44:00.860285 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.860220 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk"] Jul 17 18:44:00.863555 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:00.863532 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba643e57-5988cb888b-clfwk"] Jul 17 18:44:02.076733 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:02.076688 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940d91e8-72db-4331-8459-525e320afde5" path="/var/lib/kubelet/pods/940d91e8-72db-4331-8459-525e320afde5/volumes" Jul 17 18:44:38.524293 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.524254 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c"] Jul 17 18:44:38.524895 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.524791 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940d91e8-72db-4331-8459-525e320afde5" containerName="maas-api" Jul 17 18:44:38.524895 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.524812 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="940d91e8-72db-4331-8459-525e320afde5" containerName="maas-api" Jul 17 18:44:38.524895 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.524833 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741ebdeb-9476-45d9-81f7-91e499c1462c" containerName="revoke-keys" Jul 17 18:44:38.524895 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.524842 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="741ebdeb-9476-45d9-81f7-91e499c1462c" containerName="revoke-keys" Jul 17 18:44:38.525149 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.524921 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="940d91e8-72db-4331-8459-525e320afde5" containerName="maas-api" Jul 17 18:44:38.525149 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.524937 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="741ebdeb-9476-45d9-81f7-91e499c1462c" containerName="revoke-keys" Jul 17 18:44:38.528436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.528416 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.531490 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.531469 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5a93a056-serving-cert\"" Jul 17 18:44:38.534843 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.534814 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:44:38.548949 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.548917 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c"] Jul 17 18:44:38.580764 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.580729 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsc9r\" (UniqueName: \"kubernetes.io/projected/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-kube-api-access-qsc9r\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.580933 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.580773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.580933 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.580837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-openshift-service-ca\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.681372 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.681339 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsc9r\" (UniqueName: \"kubernetes.io/projected/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-kube-api-access-qsc9r\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.681372 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.681378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.681647 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.681424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-openshift-service-ca\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.681647 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:44:38.681552 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-serving-cert: secret "maas-api-e2e-mt-5a93a056-serving-cert" not found Jul 17 18:44:38.681647 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:44:38.681628 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls podName:f08df076-cf6b-4c1d-8fad-5539fbe1f3b7 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:39.181606169 +0000 UTC m=+2089.596344176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls") pod "maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" (UID: "f08df076-cf6b-4c1d-8fad-5539fbe1f3b7") : secret "maas-api-e2e-mt-5a93a056-serving-cert" not found Jul 17 18:44:38.682163 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.682129 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-openshift-service-ca\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:38.703485 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:38.703458 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsc9r\" (UniqueName: \"kubernetes.io/projected/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-kube-api-access-qsc9r\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:39.186378 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:39.186341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:39.188780 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:39.188760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls\") pod \"maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:39.441477 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:39.441395 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:39.597124 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:39.597096 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c"] Jul 17 18:44:39.599175 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:44:39.599148 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08df076_cf6b_4c1d_8fad_5539fbe1f3b7.slice/crio-4ab280a43f7aac8d8782c730f7c7422b52c23a701ce4d61af495078538ed3ab4 WatchSource:0}: Error finding container 4ab280a43f7aac8d8782c730f7c7422b52c23a701ce4d61af495078538ed3ab4: Status 404 returned error can't find the container with id 4ab280a43f7aac8d8782c730f7c7422b52c23a701ce4d61af495078538ed3ab4 Jul 17 18:44:40.058797 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:40.058756 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" event={"ID":"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7","Type":"ContainerStarted","Data":"1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250"} Jul 17 18:44:40.059005 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:40.058845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" event={"ID":"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7","Type":"ContainerStarted","Data":"4ab280a43f7aac8d8782c730f7c7422b52c23a701ce4d61af495078538ed3ab4"} Jul 17 18:44:40.059005 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:40.058870 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:40.080409 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:40.080362 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" podStartSLOduration=1.722020023 podStartE2EDuration="2.080347095s" totalImagesPullingTime="358.327072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:40.079789667 +0000 UTC m=+2090.494527694" watchObservedRunningTime="2026-07-17 18:44:40.080347095 +0000 UTC m=+2090.495085122" Jul 17 18:44:46.072077 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:46.072050 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:44:50.168991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:50.168856 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:44:50.173353 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:50.171146 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:44:50.264937 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:44:50.264901 2570 scope.go:117] "RemoveContainer" containerID="b67e8636ad01d4b817e06f52b109c3ad776fae6187a09403aeded9bb64ea03fa" Jul 17 18:45:00.149247 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.149201 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2"] Jul 17 18:45:00.152830 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.152808 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg"] Jul 17 18:45:00.153002 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.152946 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" Jul 17 18:45:00.155966 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.155935 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:45:00.156172 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.156154 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" Jul 17 18:45:00.161405 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.160604 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2"] Jul 17 18:45:00.163988 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.163945 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg"] Jul 17 18:45:00.165787 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.165766 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" Jul 17 18:45:00.173566 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.173544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" Jul 17 18:45:00.561885 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.561852 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2"] Jul 17 18:45:00.562625 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:45:00.562593 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c96b3a_a3e7_419d_abe9_b71d5a4d1ed3.slice/crio-3b087cb64261d60bf9e96aa7493704c99956abc2d0bee28ee22256dade5ffaa5 WatchSource:0}: Error finding container 3b087cb64261d60bf9e96aa7493704c99956abc2d0bee28ee22256dade5ffaa5: Status 404 returned error can't find the container with id 3b087cb64261d60bf9e96aa7493704c99956abc2d0bee28ee22256dade5ffaa5 Jul 17 18:45:00.577120 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:00.577097 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg"] Jul 17 18:45:00.578571 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:45:00.578542 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6b7a34_d82b_403b_a7d7_202facf26d97.slice/crio-b02c061c91cc1c2f82a9299c4b1fc1ee25ffd49aa23a302d01c05b4842e6781c WatchSource:0}: Error finding container b02c061c91cc1c2f82a9299c4b1fc1ee25ffd49aa23a302d01c05b4842e6781c: Status 404 returned error can't find the container with id b02c061c91cc1c2f82a9299c4b1fc1ee25ffd49aa23a302d01c05b4842e6781c Jul 17 18:45:01.185262 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:01.185225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" event={"ID":"7b6b7a34-d82b-403b-a7d7-202facf26d97","Type":"ContainerStarted","Data":"907ace198095ac975a22f97e3af3373ec468862f4d9cdc8a59e6f941fed7c2c1"} Jul 17 18:45:01.185730 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:01.185292 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" event={"ID":"7b6b7a34-d82b-403b-a7d7-202facf26d97","Type":"ContainerStarted","Data":"b02c061c91cc1c2f82a9299c4b1fc1ee25ffd49aa23a302d01c05b4842e6781c"} Jul 17 18:45:01.186784 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:01.186757 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" event={"ID":"77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3","Type":"ContainerStarted","Data":"b0d5fdc10eeef4e45004111be6e4c45fe6364e9859798bf1db6d5976e899a25c"} Jul 17 18:45:01.186869 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:01.186814 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" event={"ID":"77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3","Type":"ContainerStarted","Data":"3b087cb64261d60bf9e96aa7493704c99956abc2d0bee28ee22256dade5ffaa5"} Jul 17 18:45:01.201854 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:01.201725 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" podStartSLOduration=1.201706804 podStartE2EDuration="1.201706804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:01.201065514 +0000 UTC m=+2111.615803540" watchObservedRunningTime="2026-07-17 18:45:01.201706804 +0000 UTC m=+2111.616444831" Jul 17 18:45:01.221260 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:01.221205 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" podStartSLOduration=1.2211878 podStartE2EDuration="1.2211878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:01.218933893 +0000 UTC m=+2111.633671922" watchObservedRunningTime="2026-07-17 18:45:01.2211878 +0000 UTC m=+2111.635925828" Jul 17 18:45:02.194260 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:02.194221 2570 generic.go:356] "Generic (PLEG): container finished" podID="7b6b7a34-d82b-403b-a7d7-202facf26d97" containerID="907ace198095ac975a22f97e3af3373ec468862f4d9cdc8a59e6f941fed7c2c1" exitCode=0 Jul 17 18:45:02.194720 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:02.194301 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" event={"ID":"7b6b7a34-d82b-403b-a7d7-202facf26d97","Type":"ContainerDied","Data":"907ace198095ac975a22f97e3af3373ec468862f4d9cdc8a59e6f941fed7c2c1"} Jul 17 18:45:02.195811 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:02.195788 2570 generic.go:356] "Generic (PLEG): container finished" podID="77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3" containerID="b0d5fdc10eeef4e45004111be6e4c45fe6364e9859798bf1db6d5976e899a25c" exitCode=0 Jul 17 18:45:02.195918 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:02.195867 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" event={"ID":"77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3","Type":"ContainerDied","Data":"b0d5fdc10eeef4e45004111be6e4c45fe6364e9859798bf1db6d5976e899a25c"} Jul 17 18:45:03.349559 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:03.349533 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" Jul 17 18:45:03.383124 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:03.383098 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" Jul 17 18:45:04.210046 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:04.210013 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" Jul 17 18:45:04.210231 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:04.210017 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg" event={"ID":"7b6b7a34-d82b-403b-a7d7-202facf26d97","Type":"ContainerDied","Data":"b02c061c91cc1c2f82a9299c4b1fc1ee25ffd49aa23a302d01c05b4842e6781c"} Jul 17 18:45:04.210231 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:04.210134 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02c061c91cc1c2f82a9299c4b1fc1ee25ffd49aa23a302d01c05b4842e6781c" Jul 17 18:45:04.211371 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:04.211346 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" event={"ID":"77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3","Type":"ContainerDied","Data":"3b087cb64261d60bf9e96aa7493704c99956abc2d0bee28ee22256dade5ffaa5"} Jul 17 18:45:04.211371 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:04.211369 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-lkjh2" Jul 17 18:45:04.211518 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:04.211397 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b087cb64261d60bf9e96aa7493704c99956abc2d0bee28ee22256dade5ffaa5" Jul 17 18:45:26.347217 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.347185 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb"] Jul 17 18:45:26.347733 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.347551 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3" containerName="cleanup" Jul 17 18:45:26.347733 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.347563 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3" containerName="cleanup" Jul 17 18:45:26.347733 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.347578 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6b7a34-d82b-403b-a7d7-202facf26d97" containerName="cleanup" Jul 17 18:45:26.347733 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.347583 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6b7a34-d82b-403b-a7d7-202facf26d97" containerName="cleanup" Jul 17 18:45:26.347733 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.347641 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3" containerName="cleanup" Jul 17 18:45:26.347733 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.347648 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6b7a34-d82b-403b-a7d7-202facf26d97" containerName="cleanup" Jul 17 18:45:26.350932 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.350917 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" Jul 17 18:45:26.353810 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.353792 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:45:26.358881 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.358619 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb"] Jul 17 18:45:26.485841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.485800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259ecae-7656-41df-9e6b-25ac31ce116d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb\" (UID: \"4259ecae-7656-41df-9e6b-25ac31ce116d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" Jul 17 18:45:26.587259 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.587227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259ecae-7656-41df-9e6b-25ac31ce116d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb\" (UID: \"4259ecae-7656-41df-9e6b-25ac31ce116d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" Jul 17 18:45:26.587995 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.587932 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259ecae-7656-41df-9e6b-25ac31ce116d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb\" (UID: \"4259ecae-7656-41df-9e6b-25ac31ce116d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" Jul 17 18:45:26.663774 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.663691 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" Jul 17 18:45:26.810623 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:26.810591 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb"] Jul 17 18:45:26.811853 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:45:26.811826 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4259ecae_7656_41df_9e6b_25ac31ce116d.slice/crio-d50ecc3828bfcce676872d0a1178fea241c23a5e50856310330e3705998a593c WatchSource:0}: Error finding container d50ecc3828bfcce676872d0a1178fea241c23a5e50856310330e3705998a593c: Status 404 returned error can't find the container with id d50ecc3828bfcce676872d0a1178fea241c23a5e50856310330e3705998a593c Jul 17 18:45:27.353854 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:27.353760 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" event={"ID":"4259ecae-7656-41df-9e6b-25ac31ce116d","Type":"ContainerStarted","Data":"7ba8aee9086d7842a41ee3446f8619002996d4b1093ce5162ecb6b84cf62dac4"} Jul 17 18:45:27.353854 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:27.353811 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" event={"ID":"4259ecae-7656-41df-9e6b-25ac31ce116d","Type":"ContainerStarted","Data":"d50ecc3828bfcce676872d0a1178fea241c23a5e50856310330e3705998a593c"} Jul 17 18:45:27.370552 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:27.370500 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" podStartSLOduration=1.3704798409999999 podStartE2EDuration="1.370479841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:27.369949245 +0000 UTC m=+2137.784687283" watchObservedRunningTime="2026-07-17 18:45:27.370479841 +0000 UTC m=+2137.785217870" Jul 17 18:45:28.361541 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:28.361509 2570 generic.go:356] "Generic (PLEG): container finished" podID="4259ecae-7656-41df-9e6b-25ac31ce116d" containerID="7ba8aee9086d7842a41ee3446f8619002996d4b1093ce5162ecb6b84cf62dac4" exitCode=0 Jul 17 18:45:28.361992 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:28.361600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" event={"ID":"4259ecae-7656-41df-9e6b-25ac31ce116d","Type":"ContainerDied","Data":"7ba8aee9086d7842a41ee3446f8619002996d4b1093ce5162ecb6b84cf62dac4"} Jul 17 18:45:29.499572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:29.499546 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" Jul 17 18:45:29.611586 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:29.611546 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259ecae-7656-41df-9e6b-25ac31ce116d-maas-api-service-ca\") pod \"4259ecae-7656-41df-9e6b-25ac31ce116d\" (UID: \"4259ecae-7656-41df-9e6b-25ac31ce116d\") " Jul 17 18:45:29.611912 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:29.611890 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4259ecae-7656-41df-9e6b-25ac31ce116d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4259ecae-7656-41df-9e6b-25ac31ce116d" (UID: "4259ecae-7656-41df-9e6b-25ac31ce116d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:29.712708 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:29.712659 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259ecae-7656-41df-9e6b-25ac31ce116d-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:45:30.373240 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:30.373161 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" Jul 17 18:45:30.373240 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:30.373201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb" event={"ID":"4259ecae-7656-41df-9e6b-25ac31ce116d","Type":"ContainerDied","Data":"d50ecc3828bfcce676872d0a1178fea241c23a5e50856310330e3705998a593c"} Jul 17 18:45:30.373419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:30.373245 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d50ecc3828bfcce676872d0a1178fea241c23a5e50856310330e3705998a593c" Jul 17 18:45:36.828856 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:36.828818 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c"] Jul 17 18:45:36.829493 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:36.829133 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" podUID="f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" containerName="maas-api" containerID="cri-o://1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250" gracePeriod=30 Jul 17 18:45:36.857020 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:36.856988 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg"] Jul 17 18:45:36.859345 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:36.859318 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5a93a056-29738565-862cg"] Jul 17 18:45:37.087290 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.087229 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:45:37.174579 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.174544 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-openshift-service-ca\") pod \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " Jul 17 18:45:37.174760 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.174587 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsc9r\" (UniqueName: \"kubernetes.io/projected/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-kube-api-access-qsc9r\") pod \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " Jul 17 18:45:37.174760 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.174612 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls\") pod \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\" (UID: \"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7\") " Jul 17 18:45:37.175036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.175010 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" (UID: "f08df076-cf6b-4c1d-8fad-5539fbe1f3b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:37.176779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.176753 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" (UID: "f08df076-cf6b-4c1d-8fad-5539fbe1f3b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:45:37.176894 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.176786 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-kube-api-access-qsc9r" (OuterVolumeSpecName: "kube-api-access-qsc9r") pod "f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" (UID: "f08df076-cf6b-4c1d-8fad-5539fbe1f3b7"). InnerVolumeSpecName "kube-api-access-qsc9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:45:37.276030 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.275999 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsc9r\" (UniqueName: \"kubernetes.io/projected/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-kube-api-access-qsc9r\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:45:37.276030 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.276025 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:45:37.276030 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.276034 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:45:37.412929 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.412844 2570 generic.go:356] "Generic (PLEG): container finished" podID="f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" containerID="1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250" exitCode=0 Jul 17 18:45:37.412929 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.412894 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" Jul 17 18:45:37.413294 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.412926 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" event={"ID":"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7","Type":"ContainerDied","Data":"1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250"} Jul 17 18:45:37.413294 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.412991 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c" event={"ID":"f08df076-cf6b-4c1d-8fad-5539fbe1f3b7","Type":"ContainerDied","Data":"4ab280a43f7aac8d8782c730f7c7422b52c23a701ce4d61af495078538ed3ab4"} Jul 17 18:45:37.413294 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.413013 2570 scope.go:117] "RemoveContainer" containerID="1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250" Jul 17 18:45:37.424817 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.424793 2570 scope.go:117] "RemoveContainer" containerID="1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250" Jul 17 18:45:37.425154 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:45:37.425133 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250\": container with ID starting with 1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250 not found: ID does not exist" containerID="1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250" Jul 17 18:45:37.425237 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.425163 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250"} err="failed to get container status \"1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250\": rpc error: code = NotFound desc = could not find container \"1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250\": container with ID starting with 1cff5c68dd7dc6ece03d8264fe96797759c9e149ce00d92f1059324cf5a4e250 not found: ID does not exist" Jul 17 18:45:37.438286 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.438257 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c"] Jul 17 18:45:37.443544 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:37.443519 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5a93a056-5d6dc8d845-xt64c"] Jul 17 18:45:38.072622 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:38.072591 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b6b7a34-d82b-403b-a7d7-202facf26d97" path="/var/lib/kubelet/pods/7b6b7a34-d82b-403b-a7d7-202facf26d97/volumes" Jul 17 18:45:38.073021 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:38.072971 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" path="/var/lib/kubelet/pods/f08df076-cf6b-4c1d-8fad-5539fbe1f3b7/volumes" Jul 17 18:45:50.290245 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:45:50.290215 2570 scope.go:117] "RemoveContainer" containerID="e708e93b8845a4e575f73b9724ef38bc267dd91dc7935722e589aa49b9f51524" Jul 17 18:46:50.317937 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:46:50.317906 2570 scope.go:117] "RemoveContainer" containerID="61c23d787c6eb714836359e80113ed3bebc31f3983e4de2a255796ebbc90479c" Jul 17 18:47:12.456273 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.456186 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5"] Jul 17 18:47:12.456841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.456740 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4259ecae-7656-41df-9e6b-25ac31ce116d" containerName="revoke-keys" Jul 17 18:47:12.456841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.456760 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4259ecae-7656-41df-9e6b-25ac31ce116d" containerName="revoke-keys" Jul 17 18:47:12.456841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.456804 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" containerName="maas-api" Jul 17 18:47:12.456841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.456814 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" containerName="maas-api" Jul 17 18:47:12.457104 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.456884 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08df076-cf6b-4c1d-8fad-5539fbe1f3b7" containerName="maas-api" Jul 17 18:47:12.457104 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.456899 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4259ecae-7656-41df-9e6b-25ac31ce116d" containerName="revoke-keys" Jul 17 18:47:12.460091 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.460065 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.466426 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.466399 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-764e6c-serving-cert\"" Jul 17 18:47:12.466426 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.466414 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:47:12.477332 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.477307 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5"] Jul 17 18:47:12.564735 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.564694 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210fa1bb-6265-4e80-94a2-8c1b79dece06-openshift-service-ca\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.564927 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.564781 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn5x\" (UniqueName: \"kubernetes.io/projected/210fa1bb-6265-4e80-94a2-8c1b79dece06-kube-api-access-sfn5x\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.564927 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.564813 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210fa1bb-6265-4e80-94a2-8c1b79dece06-maas-api-tls\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.665621 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.665582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210fa1bb-6265-4e80-94a2-8c1b79dece06-openshift-service-ca\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.665813 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.665644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn5x\" (UniqueName: \"kubernetes.io/projected/210fa1bb-6265-4e80-94a2-8c1b79dece06-kube-api-access-sfn5x\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.665813 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.665674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210fa1bb-6265-4e80-94a2-8c1b79dece06-maas-api-tls\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.666273 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.666250 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210fa1bb-6265-4e80-94a2-8c1b79dece06-openshift-service-ca\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.668286 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.668262 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210fa1bb-6265-4e80-94a2-8c1b79dece06-maas-api-tls\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.679033 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.679005 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn5x\" (UniqueName: \"kubernetes.io/projected/210fa1bb-6265-4e80-94a2-8c1b79dece06-kube-api-access-sfn5x\") pod \"maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.770370 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.770280 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:12.921224 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.921196 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5"] Jul 17 18:47:12.923570 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:47:12.923541 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210fa1bb_6265_4e80_94a2_8c1b79dece06.slice/crio-89b878dec15120923d9f1e56e1edecfabf4483ae618af3442e45028796761e28 WatchSource:0}: Error finding container 89b878dec15120923d9f1e56e1edecfabf4483ae618af3442e45028796761e28: Status 404 returned error can't find the container with id 89b878dec15120923d9f1e56e1edecfabf4483ae618af3442e45028796761e28 Jul 17 18:47:12.953892 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:12.953855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" event={"ID":"210fa1bb-6265-4e80-94a2-8c1b79dece06","Type":"ContainerStarted","Data":"89b878dec15120923d9f1e56e1edecfabf4483ae618af3442e45028796761e28"} Jul 17 18:47:13.961710 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:13.961673 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" event={"ID":"210fa1bb-6265-4e80-94a2-8c1b79dece06","Type":"ContainerStarted","Data":"50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f"} Jul 17 18:47:13.962144 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:13.961759 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:13.980367 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:13.980316 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" podStartSLOduration=1.287243562 podStartE2EDuration="1.980298602s" totalImagesPullingTime="693.05504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:13.977772252 +0000 UTC m=+2244.392510280" watchObservedRunningTime="2026-07-17 18:47:13.980298602 +0000 UTC m=+2244.395036657" Jul 17 18:47:19.973065 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:19.973029 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:47:20.264346 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.264253 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj"] Jul 17 18:47:20.267945 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.267919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.271205 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.271184 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8f4385-serving-cert\"" Jul 17 18:47:20.316512 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.316481 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj"] Jul 17 18:47:20.336927 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.336891 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.337109 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.336932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxkn\" (UniqueName: \"kubernetes.io/projected/b7f03eba-7bea-4792-acaf-2e1982453009-kube-api-access-rtxkn\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.337109 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.337083 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f03eba-7bea-4792-acaf-2e1982453009-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.438024 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.437987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.438024 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.438026 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxkn\" (UniqueName: \"kubernetes.io/projected/b7f03eba-7bea-4792-acaf-2e1982453009-kube-api-access-rtxkn\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.438254 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.438083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f03eba-7bea-4792-acaf-2e1982453009-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.438254 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:47:20.438164 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-serving-cert: secret "maas-api-e2e-inf-b-8f4385-serving-cert" not found Jul 17 18:47:20.438343 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:47:20.438262 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls podName:b7f03eba-7bea-4792-acaf-2e1982453009 nodeName:}" failed. No retries permitted until 2026-07-17 18:47:20.938237292 +0000 UTC m=+2251.352975312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls") pod "maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" (UID: "b7f03eba-7bea-4792-acaf-2e1982453009") : secret "maas-api-e2e-inf-b-8f4385-serving-cert" not found Jul 17 18:47:20.438671 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.438655 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f03eba-7bea-4792-acaf-2e1982453009-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.457870 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.457841 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxkn\" (UniqueName: \"kubernetes.io/projected/b7f03eba-7bea-4792-acaf-2e1982453009-kube-api-access-rtxkn\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.941329 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.941294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:20.944095 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:20.944062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls\") pod \"maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:21.178841 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:21.178801 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:21.346126 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:21.346094 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj"] Jul 17 18:47:21.346747 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:47:21.346717 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f03eba_7bea_4792_acaf_2e1982453009.slice/crio-37c3bd22d47ec532d938bd4e89ea7cb642913cf0cb05a28bef1239c5cee89851 WatchSource:0}: Error finding container 37c3bd22d47ec532d938bd4e89ea7cb642913cf0cb05a28bef1239c5cee89851: Status 404 returned error can't find the container with id 37c3bd22d47ec532d938bd4e89ea7cb642913cf0cb05a28bef1239c5cee89851 Jul 17 18:47:22.011914 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:22.011882 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" event={"ID":"b7f03eba-7bea-4792-acaf-2e1982453009","Type":"ContainerStarted","Data":"b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab"} Jul 17 18:47:22.012122 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:22.011931 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" event={"ID":"b7f03eba-7bea-4792-acaf-2e1982453009","Type":"ContainerStarted","Data":"37c3bd22d47ec532d938bd4e89ea7cb642913cf0cb05a28bef1239c5cee89851"} Jul 17 18:47:22.012122 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:22.011984 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:47:22.040676 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:22.040606 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" podStartSLOduration=1.53429957 podStartE2EDuration="2.040583047s" totalImagesPullingTime="506.283477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:22.03446339 +0000 UTC m=+2252.449201419" watchObservedRunningTime="2026-07-17 18:47:22.040583047 +0000 UTC m=+2252.455321076" Jul 17 18:47:28.024050 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:47:28.024022 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:48:14.031473 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.031434 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62"] Jul 17 18:48:14.034999 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.034981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" Jul 17 18:48:14.037605 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.037585 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:48:14.045704 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.045678 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62"] Jul 17 18:48:14.092879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.092848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426e5e33-c125-4281-aa1a-c970fc804887-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62\" (UID: \"426e5e33-c125-4281-aa1a-c970fc804887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" Jul 17 18:48:14.193823 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.193786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426e5e33-c125-4281-aa1a-c970fc804887-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62\" (UID: \"426e5e33-c125-4281-aa1a-c970fc804887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" Jul 17 18:48:14.194431 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.194411 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426e5e33-c125-4281-aa1a-c970fc804887-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62\" (UID: \"426e5e33-c125-4281-aa1a-c970fc804887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" Jul 17 18:48:14.346194 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.346093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" Jul 17 18:48:14.490762 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:14.490737 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62"] Jul 17 18:48:14.492903 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:48:14.492878 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426e5e33_c125_4281_aa1a_c970fc804887.slice/crio-f1fd2e25285990fe4db7920dfc02c49d31d7adeed28287f6ac9052e89fc121c6 WatchSource:0}: Error finding container f1fd2e25285990fe4db7920dfc02c49d31d7adeed28287f6ac9052e89fc121c6: Status 404 returned error can't find the container with id f1fd2e25285990fe4db7920dfc02c49d31d7adeed28287f6ac9052e89fc121c6 Jul 17 18:48:15.335198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:15.335159 2570 generic.go:356] "Generic (PLEG): container finished" podID="426e5e33-c125-4281-aa1a-c970fc804887" containerID="aff364b1ffbfb9127a62df86f2129539599b72b37066f0041d858b87a0ca8f3b" exitCode=0 Jul 17 18:48:15.335612 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:15.335213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" event={"ID":"426e5e33-c125-4281-aa1a-c970fc804887","Type":"ContainerDied","Data":"aff364b1ffbfb9127a62df86f2129539599b72b37066f0041d858b87a0ca8f3b"} Jul 17 18:48:15.335612 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:15.335276 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" event={"ID":"426e5e33-c125-4281-aa1a-c970fc804887","Type":"ContainerStarted","Data":"f1fd2e25285990fe4db7920dfc02c49d31d7adeed28287f6ac9052e89fc121c6"} Jul 17 18:48:16.476584 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:16.476561 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" Jul 17 18:48:16.513563 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:16.513534 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426e5e33-c125-4281-aa1a-c970fc804887-maas-api-service-ca\") pod \"426e5e33-c125-4281-aa1a-c970fc804887\" (UID: \"426e5e33-c125-4281-aa1a-c970fc804887\") " Jul 17 18:48:16.513879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:16.513857 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426e5e33-c125-4281-aa1a-c970fc804887-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "426e5e33-c125-4281-aa1a-c970fc804887" (UID: "426e5e33-c125-4281-aa1a-c970fc804887"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:16.614912 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:16.614827 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426e5e33-c125-4281-aa1a-c970fc804887-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:17.347823 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:17.347792 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" Jul 17 18:48:17.348050 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:17.347828 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62" event={"ID":"426e5e33-c125-4281-aa1a-c970fc804887","Type":"ContainerDied","Data":"f1fd2e25285990fe4db7920dfc02c49d31d7adeed28287f6ac9052e89fc121c6"} Jul 17 18:48:17.348050 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:17.347872 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1fd2e25285990fe4db7920dfc02c49d31d7adeed28287f6ac9052e89fc121c6" Jul 17 18:48:24.579399 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.579361 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5"] Jul 17 18:48:24.579881 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.579685 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" podUID="210fa1bb-6265-4e80-94a2-8c1b79dece06" containerName="maas-api" containerID="cri-o://50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f" gracePeriod=30 Jul 17 18:48:24.833452 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.833387 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:48:24.886735 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.886704 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfn5x\" (UniqueName: \"kubernetes.io/projected/210fa1bb-6265-4e80-94a2-8c1b79dece06-kube-api-access-sfn5x\") pod \"210fa1bb-6265-4e80-94a2-8c1b79dece06\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " Jul 17 18:48:24.886899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.886745 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210fa1bb-6265-4e80-94a2-8c1b79dece06-openshift-service-ca\") pod \"210fa1bb-6265-4e80-94a2-8c1b79dece06\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " Jul 17 18:48:24.886899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.886795 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210fa1bb-6265-4e80-94a2-8c1b79dece06-maas-api-tls\") pod \"210fa1bb-6265-4e80-94a2-8c1b79dece06\" (UID: \"210fa1bb-6265-4e80-94a2-8c1b79dece06\") " Jul 17 18:48:24.887192 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.887163 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210fa1bb-6265-4e80-94a2-8c1b79dece06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "210fa1bb-6265-4e80-94a2-8c1b79dece06" (UID: "210fa1bb-6265-4e80-94a2-8c1b79dece06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:24.888820 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.888785 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210fa1bb-6265-4e80-94a2-8c1b79dece06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "210fa1bb-6265-4e80-94a2-8c1b79dece06" (UID: "210fa1bb-6265-4e80-94a2-8c1b79dece06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:48:24.888934 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.888837 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210fa1bb-6265-4e80-94a2-8c1b79dece06-kube-api-access-sfn5x" (OuterVolumeSpecName: "kube-api-access-sfn5x") pod "210fa1bb-6265-4e80-94a2-8c1b79dece06" (UID: "210fa1bb-6265-4e80-94a2-8c1b79dece06"). InnerVolumeSpecName "kube-api-access-sfn5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:48:24.988238 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.988197 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn5x\" (UniqueName: \"kubernetes.io/projected/210fa1bb-6265-4e80-94a2-8c1b79dece06-kube-api-access-sfn5x\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:24.988238 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.988230 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210fa1bb-6265-4e80-94a2-8c1b79dece06-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:24.988238 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:24.988240 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210fa1bb-6265-4e80-94a2-8c1b79dece06-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:25.391822 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.391785 2570 generic.go:356] "Generic (PLEG): container finished" podID="210fa1bb-6265-4e80-94a2-8c1b79dece06" containerID="50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f" exitCode=0 Jul 17 18:48:25.392086 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.391833 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" Jul 17 18:48:25.392086 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.391866 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" event={"ID":"210fa1bb-6265-4e80-94a2-8c1b79dece06","Type":"ContainerDied","Data":"50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f"} Jul 17 18:48:25.392086 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.391920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5" event={"ID":"210fa1bb-6265-4e80-94a2-8c1b79dece06","Type":"ContainerDied","Data":"89b878dec15120923d9f1e56e1edecfabf4483ae618af3442e45028796761e28"} Jul 17 18:48:25.392086 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.391943 2570 scope.go:117] "RemoveContainer" containerID="50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f" Jul 17 18:48:25.401780 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.401759 2570 scope.go:117] "RemoveContainer" containerID="50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f" Jul 17 18:48:25.402050 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:48:25.402032 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f\": container with ID starting with 50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f not found: ID does not exist" containerID="50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f" Jul 17 18:48:25.402103 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.402060 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f"} err="failed to get container status \"50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f\": rpc error: code = NotFound desc = could not find container \"50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f\": container with ID starting with 50bf071a54a6156409669879d07084214d7858ebde7ab677283213356457283f not found: ID does not exist" Jul 17 18:48:25.415703 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.415653 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5"] Jul 17 18:48:25.419051 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:25.419022 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-764e6c-69d6665686-q6nc5"] Jul 17 18:48:26.072200 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:26.072158 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="210fa1bb-6265-4e80-94a2-8c1b79dece06" path="/var/lib/kubelet/pods/210fa1bb-6265-4e80-94a2-8c1b79dece06/volumes" Jul 17 18:48:40.265568 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.265532 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh"] Jul 17 18:48:40.266019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.265896 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210fa1bb-6265-4e80-94a2-8c1b79dece06" containerName="maas-api" Jul 17 18:48:40.266019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.265907 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="210fa1bb-6265-4e80-94a2-8c1b79dece06" containerName="maas-api" Jul 17 18:48:40.266019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.265921 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426e5e33-c125-4281-aa1a-c970fc804887" containerName="revoke-keys" Jul 17 18:48:40.266019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.265927 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="426e5e33-c125-4281-aa1a-c970fc804887" containerName="revoke-keys" Jul 17 18:48:40.266019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.265995 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="210fa1bb-6265-4e80-94a2-8c1b79dece06" containerName="maas-api" Jul 17 18:48:40.266019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.266005 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="426e5e33-c125-4281-aa1a-c970fc804887" containerName="revoke-keys" Jul 17 18:48:40.270495 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.270479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" Jul 17 18:48:40.274361 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.274335 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:48:40.289392 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.289368 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh"] Jul 17 18:48:40.427199 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.427160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab52254d-cd30-47db-86be-3098345ac7fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh\" (UID: \"ab52254d-cd30-47db-86be-3098345ac7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" Jul 17 18:48:40.528207 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.528119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab52254d-cd30-47db-86be-3098345ac7fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh\" (UID: \"ab52254d-cd30-47db-86be-3098345ac7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" Jul 17 18:48:40.528728 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.528708 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab52254d-cd30-47db-86be-3098345ac7fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh\" (UID: \"ab52254d-cd30-47db-86be-3098345ac7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" Jul 17 18:48:40.580209 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.580170 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" Jul 17 18:48:40.729785 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:40.729759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh"] Jul 17 18:48:40.732549 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:48:40.732519 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab52254d_cd30_47db_86be_3098345ac7fd.slice/crio-8bfb2a54ea0ac6d084b958024157faa577320d8f34043573069a8ce2d7c33eb5 WatchSource:0}: Error finding container 8bfb2a54ea0ac6d084b958024157faa577320d8f34043573069a8ce2d7c33eb5: Status 404 returned error can't find the container with id 8bfb2a54ea0ac6d084b958024157faa577320d8f34043573069a8ce2d7c33eb5 Jul 17 18:48:41.484693 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:41.484664 2570 generic.go:356] "Generic (PLEG): container finished" podID="ab52254d-cd30-47db-86be-3098345ac7fd" containerID="c917c241a36ab56aeb2298096a71748a84b5964145201d1015de09100805b2a8" exitCode=0 Jul 17 18:48:41.485095 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:41.484749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" event={"ID":"ab52254d-cd30-47db-86be-3098345ac7fd","Type":"ContainerDied","Data":"c917c241a36ab56aeb2298096a71748a84b5964145201d1015de09100805b2a8"} Jul 17 18:48:41.485095 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:41.484803 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" event={"ID":"ab52254d-cd30-47db-86be-3098345ac7fd","Type":"ContainerStarted","Data":"8bfb2a54ea0ac6d084b958024157faa577320d8f34043573069a8ce2d7c33eb5"} Jul 17 18:48:42.631307 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:42.631283 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" Jul 17 18:48:42.749793 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:42.749710 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab52254d-cd30-47db-86be-3098345ac7fd-maas-api-service-ca\") pod \"ab52254d-cd30-47db-86be-3098345ac7fd\" (UID: \"ab52254d-cd30-47db-86be-3098345ac7fd\") " Jul 17 18:48:42.750042 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:42.750020 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab52254d-cd30-47db-86be-3098345ac7fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab52254d-cd30-47db-86be-3098345ac7fd" (UID: "ab52254d-cd30-47db-86be-3098345ac7fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:42.850991 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:42.850920 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab52254d-cd30-47db-86be-3098345ac7fd-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:43.497860 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:43.497820 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" event={"ID":"ab52254d-cd30-47db-86be-3098345ac7fd","Type":"ContainerDied","Data":"8bfb2a54ea0ac6d084b958024157faa577320d8f34043573069a8ce2d7c33eb5"} Jul 17 18:48:43.497860 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:43.497855 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh" Jul 17 18:48:43.498100 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:43.497880 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bfb2a54ea0ac6d084b958024157faa577320d8f34043573069a8ce2d7c33eb5" Jul 17 18:48:50.893890 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:50.893855 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj"] Jul 17 18:48:50.894312 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:50.894080 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" podUID="b7f03eba-7bea-4792-acaf-2e1982453009" containerName="maas-api" containerID="cri-o://b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab" gracePeriod=30 Jul 17 18:48:51.167132 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.167110 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:48:51.233197 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.233165 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f03eba-7bea-4792-acaf-2e1982453009-openshift-service-ca\") pod \"b7f03eba-7bea-4792-acaf-2e1982453009\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " Jul 17 18:48:51.233367 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.233206 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtxkn\" (UniqueName: \"kubernetes.io/projected/b7f03eba-7bea-4792-acaf-2e1982453009-kube-api-access-rtxkn\") pod \"b7f03eba-7bea-4792-acaf-2e1982453009\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " Jul 17 18:48:51.233594 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.233560 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f03eba-7bea-4792-acaf-2e1982453009-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7f03eba-7bea-4792-acaf-2e1982453009" (UID: "b7f03eba-7bea-4792-acaf-2e1982453009"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:51.235235 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.235214 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f03eba-7bea-4792-acaf-2e1982453009-kube-api-access-rtxkn" (OuterVolumeSpecName: "kube-api-access-rtxkn") pod "b7f03eba-7bea-4792-acaf-2e1982453009" (UID: "b7f03eba-7bea-4792-acaf-2e1982453009"). InnerVolumeSpecName "kube-api-access-rtxkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:48:51.333607 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.333567 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls\") pod \"b7f03eba-7bea-4792-acaf-2e1982453009\" (UID: \"b7f03eba-7bea-4792-acaf-2e1982453009\") " Jul 17 18:48:51.333814 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.333740 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f03eba-7bea-4792-acaf-2e1982453009-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:51.333814 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.333752 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtxkn\" (UniqueName: \"kubernetes.io/projected/b7f03eba-7bea-4792-acaf-2e1982453009-kube-api-access-rtxkn\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:51.335654 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.335616 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7f03eba-7bea-4792-acaf-2e1982453009" (UID: "b7f03eba-7bea-4792-acaf-2e1982453009"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:48:51.434753 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.434721 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7f03eba-7bea-4792-acaf-2e1982453009-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:48:51.546615 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.546581 2570 generic.go:356] "Generic (PLEG): container finished" podID="b7f03eba-7bea-4792-acaf-2e1982453009" containerID="b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab" exitCode=0 Jul 17 18:48:51.546787 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.546629 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" Jul 17 18:48:51.546787 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.546660 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" event={"ID":"b7f03eba-7bea-4792-acaf-2e1982453009","Type":"ContainerDied","Data":"b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab"} Jul 17 18:48:51.546787 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.546720 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj" event={"ID":"b7f03eba-7bea-4792-acaf-2e1982453009","Type":"ContainerDied","Data":"37c3bd22d47ec532d938bd4e89ea7cb642913cf0cb05a28bef1239c5cee89851"} Jul 17 18:48:51.546787 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.546743 2570 scope.go:117] "RemoveContainer" containerID="b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab" Jul 17 18:48:51.556459 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.556438 2570 scope.go:117] "RemoveContainer" containerID="b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab" Jul 17 18:48:51.556747 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:48:51.556730 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab\": container with ID starting with b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab not found: ID does not exist" containerID="b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab" Jul 17 18:48:51.556795 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.556758 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab"} err="failed to get container status \"b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab\": rpc error: code = NotFound desc = could not find container \"b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab\": container with ID starting with b0342a7e62c41183f0b8882a3202ee08a271e7b26be2b118a379800a687e04ab not found: ID does not exist" Jul 17 18:48:51.574046 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.574021 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj"] Jul 17 18:48:51.578894 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:51.578868 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f4385-78b65f6fb4-kfmvj"] Jul 17 18:48:52.073096 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:48:52.073060 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f03eba-7bea-4792-acaf-2e1982453009" path="/var/lib/kubelet/pods/b7f03eba-7bea-4792-acaf-2e1982453009/volumes" Jul 17 18:49:09.933022 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.932977 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4"] Jul 17 18:49:09.933506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.933337 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f03eba-7bea-4792-acaf-2e1982453009" containerName="maas-api" Jul 17 18:49:09.933506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.933347 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f03eba-7bea-4792-acaf-2e1982453009" containerName="maas-api" Jul 17 18:49:09.933506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.933357 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab52254d-cd30-47db-86be-3098345ac7fd" containerName="revoke-keys" Jul 17 18:49:09.933506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.933362 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab52254d-cd30-47db-86be-3098345ac7fd" containerName="revoke-keys" Jul 17 18:49:09.933506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.933419 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab52254d-cd30-47db-86be-3098345ac7fd" containerName="revoke-keys" Jul 17 18:49:09.933506 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.933429 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f03eba-7bea-4792-acaf-2e1982453009" containerName="maas-api" Jul 17 18:49:09.936592 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.936570 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:09.940699 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.940676 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-82b1be-serving-cert\"" Jul 17 18:49:09.942050 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.942032 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:49:09.955716 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.955682 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4"] Jul 17 18:49:09.987888 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.987855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:09.988106 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.987903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c92285-0e4a-4029-8bea-930c36608ae2-openshift-service-ca\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:09.988106 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:09.987939 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj425\" (UniqueName: \"kubernetes.io/projected/07c92285-0e4a-4029-8bea-930c36608ae2-kube-api-access-mj425\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.089313 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.089283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.089313 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.089325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c92285-0e4a-4029-8bea-930c36608ae2-openshift-service-ca\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.089545 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.089356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj425\" (UniqueName: \"kubernetes.io/projected/07c92285-0e4a-4029-8bea-930c36608ae2-kube-api-access-mj425\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.089545 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:49:10.089443 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-serving-cert: secret "maas-api-e2e-api-a-82b1be-serving-cert" not found Jul 17 18:49:10.089545 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:49:10.089516 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls podName:07c92285-0e4a-4029-8bea-930c36608ae2 nodeName:}" failed. No retries permitted until 2026-07-17 18:49:10.589499461 +0000 UTC m=+2361.004237465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls") pod "maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" (UID: "07c92285-0e4a-4029-8bea-930c36608ae2") : secret "maas-api-e2e-api-a-82b1be-serving-cert" not found Jul 17 18:49:10.090030 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.090013 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c92285-0e4a-4029-8bea-930c36608ae2-openshift-service-ca\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.102533 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.102507 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj425\" (UniqueName: \"kubernetes.io/projected/07c92285-0e4a-4029-8bea-930c36608ae2-kube-api-access-mj425\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.594423 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.594381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.596774 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.596753 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls\") pod \"maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.849420 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.849330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:10.989947 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.989910 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4"] Jul 17 18:49:10.990404 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:49:10.990382 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c92285_0e4a_4029_8bea_930c36608ae2.slice/crio-d04a0e5a0cb51639b3ed70a7e9dde30272304dee74d2afb913a112fca072c12a WatchSource:0}: Error finding container d04a0e5a0cb51639b3ed70a7e9dde30272304dee74d2afb913a112fca072c12a: Status 404 returned error can't find the container with id d04a0e5a0cb51639b3ed70a7e9dde30272304dee74d2afb913a112fca072c12a Jul 17 18:49:10.992001 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:10.991985 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:49:11.666128 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:11.666082 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" event={"ID":"07c92285-0e4a-4029-8bea-930c36608ae2","Type":"ContainerStarted","Data":"d04a0e5a0cb51639b3ed70a7e9dde30272304dee74d2afb913a112fca072c12a"} Jul 17 18:49:12.674499 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:12.674462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" event={"ID":"07c92285-0e4a-4029-8bea-930c36608ae2","Type":"ContainerStarted","Data":"4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509"} Jul 17 18:49:12.675001 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:12.674573 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:17.875044 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.874989 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" podStartSLOduration=8.109192158 podStartE2EDuration="8.874949269s" totalImagesPullingTime="765.757111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:49:12.708326319 +0000 UTC m=+2363.123064345" watchObservedRunningTime="2026-07-17 18:49:17.874949269 +0000 UTC m=+2368.289687296" Jul 17 18:49:17.877092 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.877063 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6"] Jul 17 18:49:17.881007 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.880988 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:17.883897 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.883874 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2cfeb4-serving-cert\"" Jul 17 18:49:17.891362 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.891338 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6"] Jul 17 18:49:17.959200 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.959158 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-openshift-service-ca\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:17.959380 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.959259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhrw\" (UniqueName: \"kubernetes.io/projected/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-kube-api-access-xzhrw\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:17.959380 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:17.959307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.060485 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.060441 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.060701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.060531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-openshift-service-ca\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.060701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.060581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhrw\" (UniqueName: \"kubernetes.io/projected/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-kube-api-access-xzhrw\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.060701 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:49:18.060599 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-serving-cert: secret "maas-api-e2e-api-b-2cfeb4-serving-cert" not found Jul 17 18:49:18.060701 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:49:18.060686 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls podName:60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce nodeName:}" failed. No retries permitted until 2026-07-17 18:49:18.56066298 +0000 UTC m=+2368.975401204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls") pod "maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" (UID: "60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce") : secret "maas-api-e2e-api-b-2cfeb4-serving-cert" not found Jul 17 18:49:18.061198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.061180 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-openshift-service-ca\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.077532 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.077507 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhrw\" (UniqueName: \"kubernetes.io/projected/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-kube-api-access-xzhrw\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.564858 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.564811 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.567397 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.567365 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls\") pod \"maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.686040 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.686011 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:18.793296 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.793265 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:18.932507 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:18.932481 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6"] Jul 17 18:49:18.933479 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:49:18.933452 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ab77d6_d2c8_4a1a_b7aa_4cb97cec24ce.slice/crio-706aa2f9d4bc655fece151d76540ef31f489de4956b97e72c95af5f4c20732be WatchSource:0}: Error finding container 706aa2f9d4bc655fece151d76540ef31f489de4956b97e72c95af5f4c20732be: Status 404 returned error can't find the container with id 706aa2f9d4bc655fece151d76540ef31f489de4956b97e72c95af5f4c20732be Jul 17 18:49:19.729811 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:19.729765 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" event={"ID":"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce","Type":"ContainerStarted","Data":"eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d"} Jul 17 18:49:19.730034 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:19.729884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" event={"ID":"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce","Type":"ContainerStarted","Data":"706aa2f9d4bc655fece151d76540ef31f489de4956b97e72c95af5f4c20732be"} Jul 17 18:49:19.731779 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:19.730778 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:19.746590 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:19.746511 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" podStartSLOduration=2.085514209 podStartE2EDuration="2.746491363s" totalImagesPullingTime="660.977154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:49:19.746162148 +0000 UTC m=+2370.160900174" watchObservedRunningTime="2026-07-17 18:49:19.746491363 +0000 UTC m=+2370.161229394" Jul 17 18:49:26.745637 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:26.745599 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:49:30.236404 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.236372 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8"] Jul 17 18:49:30.239874 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.239853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" Jul 17 18:49:30.242520 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.242496 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:49:30.247145 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.247096 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8"] Jul 17 18:49:30.370348 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.370313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ff674-2c68-4073-9e9e-ef186afcac0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82b1be-942r8\" (UID: \"279ff674-2c68-4073-9e9e-ef186afcac0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" Jul 17 18:49:30.471646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.471609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ff674-2c68-4073-9e9e-ef186afcac0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82b1be-942r8\" (UID: \"279ff674-2c68-4073-9e9e-ef186afcac0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" Jul 17 18:49:30.472370 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.472343 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ff674-2c68-4073-9e9e-ef186afcac0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82b1be-942r8\" (UID: \"279ff674-2c68-4073-9e9e-ef186afcac0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" Jul 17 18:49:30.551610 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.551521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" Jul 17 18:49:30.689793 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.689767 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8"] Jul 17 18:49:30.691242 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:49:30.691219 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279ff674_2c68_4073_9e9e_ef186afcac0e.slice/crio-2613d5f4ffd9b55206cf4273bc5ff26511dc6abd98839dd44fc72d8c2ff7db24 WatchSource:0}: Error finding container 2613d5f4ffd9b55206cf4273bc5ff26511dc6abd98839dd44fc72d8c2ff7db24: Status 404 returned error can't find the container with id 2613d5f4ffd9b55206cf4273bc5ff26511dc6abd98839dd44fc72d8c2ff7db24 Jul 17 18:49:30.797285 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:30.797248 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" event={"ID":"279ff674-2c68-4073-9e9e-ef186afcac0e","Type":"ContainerStarted","Data":"2613d5f4ffd9b55206cf4273bc5ff26511dc6abd98839dd44fc72d8c2ff7db24"} Jul 17 18:49:31.805080 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:31.804988 2570 generic.go:356] "Generic (PLEG): container finished" podID="279ff674-2c68-4073-9e9e-ef186afcac0e" containerID="07387cd1c1409f082a3eaa140ce4fe024f057fc0662ff7696d241b1b3dc206f5" exitCode=0 Jul 17 18:49:31.805456 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:31.805124 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" event={"ID":"279ff674-2c68-4073-9e9e-ef186afcac0e","Type":"ContainerDied","Data":"07387cd1c1409f082a3eaa140ce4fe024f057fc0662ff7696d241b1b3dc206f5"} Jul 17 18:49:32.947540 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:32.947512 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" Jul 17 18:49:33.093598 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:33.093513 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ff674-2c68-4073-9e9e-ef186afcac0e-maas-api-service-ca\") pod \"279ff674-2c68-4073-9e9e-ef186afcac0e\" (UID: \"279ff674-2c68-4073-9e9e-ef186afcac0e\") " Jul 17 18:49:33.093909 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:33.093887 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279ff674-2c68-4073-9e9e-ef186afcac0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "279ff674-2c68-4073-9e9e-ef186afcac0e" (UID: "279ff674-2c68-4073-9e9e-ef186afcac0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:33.194761 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:33.194720 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ff674-2c68-4073-9e9e-ef186afcac0e-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:49:33.817630 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:33.817599 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" Jul 17 18:49:33.817803 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:33.817599 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82b1be-942r8" event={"ID":"279ff674-2c68-4073-9e9e-ef186afcac0e","Type":"ContainerDied","Data":"2613d5f4ffd9b55206cf4273bc5ff26511dc6abd98839dd44fc72d8c2ff7db24"} Jul 17 18:49:33.817803 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:33.817719 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2613d5f4ffd9b55206cf4273bc5ff26511dc6abd98839dd44fc72d8c2ff7db24" Jul 17 18:49:41.054767 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.054725 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4"] Jul 17 18:49:41.055318 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.055053 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" podUID="07c92285-0e4a-4029-8bea-930c36608ae2" containerName="maas-api" containerID="cri-o://4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509" gracePeriod=30 Jul 17 18:49:41.303645 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.303616 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:41.467523 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.467475 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj425\" (UniqueName: \"kubernetes.io/projected/07c92285-0e4a-4029-8bea-930c36608ae2-kube-api-access-mj425\") pod \"07c92285-0e4a-4029-8bea-930c36608ae2\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " Jul 17 18:49:41.467523 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.467526 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls\") pod \"07c92285-0e4a-4029-8bea-930c36608ae2\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " Jul 17 18:49:41.467797 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.467554 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c92285-0e4a-4029-8bea-930c36608ae2-openshift-service-ca\") pod \"07c92285-0e4a-4029-8bea-930c36608ae2\" (UID: \"07c92285-0e4a-4029-8bea-930c36608ae2\") " Jul 17 18:49:41.467985 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.467944 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c92285-0e4a-4029-8bea-930c36608ae2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07c92285-0e4a-4029-8bea-930c36608ae2" (UID: "07c92285-0e4a-4029-8bea-930c36608ae2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:41.469672 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.469646 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07c92285-0e4a-4029-8bea-930c36608ae2" (UID: "07c92285-0e4a-4029-8bea-930c36608ae2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:49:41.469804 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.469746 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c92285-0e4a-4029-8bea-930c36608ae2-kube-api-access-mj425" (OuterVolumeSpecName: "kube-api-access-mj425") pod "07c92285-0e4a-4029-8bea-930c36608ae2" (UID: "07c92285-0e4a-4029-8bea-930c36608ae2"). InnerVolumeSpecName "kube-api-access-mj425". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:49:41.568846 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.568812 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c92285-0e4a-4029-8bea-930c36608ae2-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:49:41.568846 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.568841 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c92285-0e4a-4029-8bea-930c36608ae2-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:49:41.568846 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.568852 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj425\" (UniqueName: \"kubernetes.io/projected/07c92285-0e4a-4029-8bea-930c36608ae2-kube-api-access-mj425\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:49:41.864359 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.864268 2570 generic.go:356] "Generic (PLEG): container finished" podID="07c92285-0e4a-4029-8bea-930c36608ae2" containerID="4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509" exitCode=0 Jul 17 18:49:41.864359 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.864315 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" Jul 17 18:49:41.864359 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.864334 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" event={"ID":"07c92285-0e4a-4029-8bea-930c36608ae2","Type":"ContainerDied","Data":"4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509"} Jul 17 18:49:41.864599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.864383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4" event={"ID":"07c92285-0e4a-4029-8bea-930c36608ae2","Type":"ContainerDied","Data":"d04a0e5a0cb51639b3ed70a7e9dde30272304dee74d2afb913a112fca072c12a"} Jul 17 18:49:41.864599 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.864411 2570 scope.go:117] "RemoveContainer" containerID="4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509" Jul 17 18:49:41.875025 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.874923 2570 scope.go:117] "RemoveContainer" containerID="4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509" Jul 17 18:49:41.877153 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:49:41.877128 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509\": container with ID starting with 4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509 not found: ID does not exist" containerID="4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509" Jul 17 18:49:41.877261 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.877157 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509"} err="failed to get container status \"4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509\": rpc error: code = NotFound desc = could not find container \"4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509\": container with ID starting with 4623236da2120e2a956f4e38f0afb25373656ca6b226edb9c7f6408c3e348509 not found: ID does not exist" Jul 17 18:49:41.889534 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.889510 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4"] Jul 17 18:49:41.893191 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:41.893166 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82b1be-579bc5f4bd-wwwm4"] Jul 17 18:49:42.072858 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:42.072823 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c92285-0e4a-4029-8bea-930c36608ae2" path="/var/lib/kubelet/pods/07c92285-0e4a-4029-8bea-930c36608ae2/volumes" Jul 17 18:49:50.208884 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:50.208858 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:49:50.215993 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:50.215949 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:49:56.351818 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.351788 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9"] Jul 17 18:49:56.352263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.352195 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279ff674-2c68-4073-9e9e-ef186afcac0e" containerName="revoke-keys" Jul 17 18:49:56.352263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.352208 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="279ff674-2c68-4073-9e9e-ef186afcac0e" containerName="revoke-keys" Jul 17 18:49:56.352263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.352229 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c92285-0e4a-4029-8bea-930c36608ae2" containerName="maas-api" Jul 17 18:49:56.352263 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.352239 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c92285-0e4a-4029-8bea-930c36608ae2" containerName="maas-api" Jul 17 18:49:56.352409 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.352305 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c92285-0e4a-4029-8bea-930c36608ae2" containerName="maas-api" Jul 17 18:49:56.352409 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.352318 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="279ff674-2c68-4073-9e9e-ef186afcac0e" containerName="revoke-keys" Jul 17 18:49:56.355338 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.355318 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" Jul 17 18:49:56.358158 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.358133 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:49:56.363244 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.363224 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9"] Jul 17 18:49:56.393607 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.393577 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337e7c03-8cb0-4052-b5e0-6dfebac50ba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9\" (UID: \"337e7c03-8cb0-4052-b5e0-6dfebac50ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" Jul 17 18:49:56.494607 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.494569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337e7c03-8cb0-4052-b5e0-6dfebac50ba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9\" (UID: \"337e7c03-8cb0-4052-b5e0-6dfebac50ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" Jul 17 18:49:56.495334 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.495307 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337e7c03-8cb0-4052-b5e0-6dfebac50ba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9\" (UID: \"337e7c03-8cb0-4052-b5e0-6dfebac50ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" Jul 17 18:49:56.666630 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.666540 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" Jul 17 18:49:56.808749 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.808716 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9"] Jul 17 18:49:56.810257 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:49:56.810230 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337e7c03_8cb0_4052_b5e0_6dfebac50ba8.slice/crio-9eda521bd4e13e856e42347fca785d112de4ec421741ae279c62f6acf15fc71d WatchSource:0}: Error finding container 9eda521bd4e13e856e42347fca785d112de4ec421741ae279c62f6acf15fc71d: Status 404 returned error can't find the container with id 9eda521bd4e13e856e42347fca785d112de4ec421741ae279c62f6acf15fc71d Jul 17 18:49:56.954655 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:56.954620 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" event={"ID":"337e7c03-8cb0-4052-b5e0-6dfebac50ba8","Type":"ContainerStarted","Data":"9eda521bd4e13e856e42347fca785d112de4ec421741ae279c62f6acf15fc71d"} Jul 17 18:49:57.961518 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:57.961475 2570 generic.go:356] "Generic (PLEG): container finished" podID="337e7c03-8cb0-4052-b5e0-6dfebac50ba8" containerID="d4d2d4dbd140657cd77bece0006a0cb96c42343f646df75333abee00e2be78a9" exitCode=0 Jul 17 18:49:57.962153 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:57.961561 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" event={"ID":"337e7c03-8cb0-4052-b5e0-6dfebac50ba8","Type":"ContainerDied","Data":"d4d2d4dbd140657cd77bece0006a0cb96c42343f646df75333abee00e2be78a9"} Jul 17 18:49:59.099546 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:59.099522 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" Jul 17 18:49:59.120917 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:59.120885 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337e7c03-8cb0-4052-b5e0-6dfebac50ba8-maas-api-service-ca\") pod \"337e7c03-8cb0-4052-b5e0-6dfebac50ba8\" (UID: \"337e7c03-8cb0-4052-b5e0-6dfebac50ba8\") " Jul 17 18:49:59.121762 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:59.121726 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337e7c03-8cb0-4052-b5e0-6dfebac50ba8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "337e7c03-8cb0-4052-b5e0-6dfebac50ba8" (UID: "337e7c03-8cb0-4052-b5e0-6dfebac50ba8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:59.221870 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:59.221779 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337e7c03-8cb0-4052-b5e0-6dfebac50ba8-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:49:59.974689 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:59.974651 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" event={"ID":"337e7c03-8cb0-4052-b5e0-6dfebac50ba8","Type":"ContainerDied","Data":"9eda521bd4e13e856e42347fca785d112de4ec421741ae279c62f6acf15fc71d"} Jul 17 18:49:59.974689 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:59.974686 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9" Jul 17 18:49:59.974910 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:49:59.974712 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eda521bd4e13e856e42347fca785d112de4ec421741ae279c62f6acf15fc71d" Jul 17 18:50:07.270812 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.270751 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6"] Jul 17 18:50:07.271307 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.271147 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" podUID="60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" containerName="maas-api" containerID="cri-o://eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d" gracePeriod=30 Jul 17 18:50:07.520090 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.520062 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:50:07.597170 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.597086 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls\") pod \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " Jul 17 18:50:07.597170 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.597148 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzhrw\" (UniqueName: \"kubernetes.io/projected/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-kube-api-access-xzhrw\") pod \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " Jul 17 18:50:07.597170 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.597169 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-openshift-service-ca\") pod \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\" (UID: \"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce\") " Jul 17 18:50:07.597538 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.597516 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" (UID: "60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:50:07.599241 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.599215 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-kube-api-access-xzhrw" (OuterVolumeSpecName: "kube-api-access-xzhrw") pod "60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" (UID: "60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce"). InnerVolumeSpecName "kube-api-access-xzhrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:50:07.599337 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.599254 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" (UID: "60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:50:07.697655 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.697615 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:50:07.697655 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.697645 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhrw\" (UniqueName: \"kubernetes.io/projected/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-kube-api-access-xzhrw\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:50:07.697655 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:07.697656 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:50:08.022368 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.022337 2570 generic.go:356] "Generic (PLEG): container finished" podID="60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" containerID="eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d" exitCode=0 Jul 17 18:50:08.022537 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.022380 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" Jul 17 18:50:08.022537 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.022418 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" event={"ID":"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce","Type":"ContainerDied","Data":"eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d"} Jul 17 18:50:08.022537 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.022475 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6" event={"ID":"60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce","Type":"ContainerDied","Data":"706aa2f9d4bc655fece151d76540ef31f489de4956b97e72c95af5f4c20732be"} Jul 17 18:50:08.022537 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.022500 2570 scope.go:117] "RemoveContainer" containerID="eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d" Jul 17 18:50:08.035732 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.035712 2570 scope.go:117] "RemoveContainer" containerID="eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d" Jul 17 18:50:08.036010 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:50:08.035990 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d\": container with ID starting with eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d not found: ID does not exist" containerID="eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d" Jul 17 18:50:08.036087 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.036024 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d"} err="failed to get container status \"eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d\": rpc error: code = NotFound desc = could not find container \"eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d\": container with ID starting with eb5af677c18ab9c17fb41639b5407b86fefa7a03403ba306e89b823b2f6c627d not found: ID does not exist" Jul 17 18:50:08.046678 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.046654 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6"] Jul 17 18:50:08.050478 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.050458 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cfeb4-54c65646d9-qmtw6"] Jul 17 18:50:08.073424 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:08.073391 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" path="/var/lib/kubelet/pods/60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce/volumes" Jul 17 18:50:26.183458 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.183419 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7"] Jul 17 18:50:26.184622 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.184592 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" containerName="maas-api" Jul 17 18:50:26.184622 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.184622 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" containerName="maas-api" Jul 17 18:50:26.184812 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.184644 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337e7c03-8cb0-4052-b5e0-6dfebac50ba8" containerName="revoke-keys" Jul 17 18:50:26.184812 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.184654 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="337e7c03-8cb0-4052-b5e0-6dfebac50ba8" containerName="revoke-keys" Jul 17 18:50:26.184812 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.184734 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ab77d6-d2c8-4a1a-b7aa-4cb97cec24ce" containerName="maas-api" Jul 17 18:50:26.184812 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.184750 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="337e7c03-8cb0-4052-b5e0-6dfebac50ba8" containerName="revoke-keys" Jul 17 18:50:26.188575 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.188552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.191561 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.191542 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksd6p\"" Jul 17 18:50:26.191717 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.191703 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4280a6-serving-cert\"" Jul 17 18:50:26.201051 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.201016 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7"] Jul 17 18:50:26.370691 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.370656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.370875 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.370746 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/2140c468-aab8-4959-93ff-1a78d1848991-kube-api-access-dpxq4\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.370875 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.370823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140c468-aab8-4959-93ff-1a78d1848991-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.472218 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.472115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/2140c468-aab8-4959-93ff-1a78d1848991-kube-api-access-dpxq4\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.472218 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.472207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140c468-aab8-4959-93ff-1a78d1848991-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.472500 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.472250 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.472500 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:50:26.472379 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-serving-cert: secret "maas-api-e2e-shared-a-4280a6-serving-cert" not found Jul 17 18:50:26.472500 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:50:26.472457 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls podName:2140c468-aab8-4959-93ff-1a78d1848991 nodeName:}" failed. No retries permitted until 2026-07-17 18:50:26.972436484 +0000 UTC m=+2437.387174515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls") pod "maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" (UID: "2140c468-aab8-4959-93ff-1a78d1848991") : secret "maas-api-e2e-shared-a-4280a6-serving-cert" not found Jul 17 18:50:26.472865 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.472845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140c468-aab8-4959-93ff-1a78d1848991-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.482196 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.482164 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/2140c468-aab8-4959-93ff-1a78d1848991-kube-api-access-dpxq4\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.976297 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.976260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:26.978700 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:26.978674 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls\") pod \"maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:27.103981 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:27.103912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:27.272530 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:27.272497 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7"] Jul 17 18:50:27.274043 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:50:27.274018 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2140c468_aab8_4959_93ff_1a78d1848991.slice/crio-0235fb64c4a726a46c9002e94bfe711ffaf227343ffcd9ae2427ebbc8d825dbe WatchSource:0}: Error finding container 0235fb64c4a726a46c9002e94bfe711ffaf227343ffcd9ae2427ebbc8d825dbe: Status 404 returned error can't find the container with id 0235fb64c4a726a46c9002e94bfe711ffaf227343ffcd9ae2427ebbc8d825dbe Jul 17 18:50:28.155523 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:28.155476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" event={"ID":"2140c468-aab8-4959-93ff-1a78d1848991","Type":"ContainerStarted","Data":"4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382"} Jul 17 18:50:28.155725 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:28.155551 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:28.155725 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:28.155573 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" event={"ID":"2140c468-aab8-4959-93ff-1a78d1848991","Type":"ContainerStarted","Data":"0235fb64c4a726a46c9002e94bfe711ffaf227343ffcd9ae2427ebbc8d825dbe"} Jul 17 18:50:28.174619 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:28.174565 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" podStartSLOduration=1.8700946809999999 podStartE2EDuration="2.174550719s" totalImagesPullingTime="304.456038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:50:28.17309366 +0000 UTC m=+2438.587831688" watchObservedRunningTime="2026-07-17 18:50:28.174550719 +0000 UTC m=+2438.589288746" Jul 17 18:50:34.168503 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.168472 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:50:34.312427 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.312387 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx"] Jul 17 18:50:34.316678 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.316650 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.320283 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.320253 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1671a0-serving-cert\"" Jul 17 18:50:34.330324 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.330291 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx"] Jul 17 18:50:34.442489 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.442433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25n45\" (UniqueName: \"kubernetes.io/projected/ea5057ea-9ded-470c-8992-4001cbfd7d15-kube-api-access-25n45\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.442670 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.442522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.442670 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.442546 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5057ea-9ded-470c-8992-4001cbfd7d15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.543121 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.543083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25n45\" (UniqueName: \"kubernetes.io/projected/ea5057ea-9ded-470c-8992-4001cbfd7d15-kube-api-access-25n45\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.543307 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.543156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.543307 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.543190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5057ea-9ded-470c-8992-4001cbfd7d15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.543397 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:50:34.543335 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-serving-cert: secret "maas-api-e2e-shared-b-1671a0-serving-cert" not found Jul 17 18:50:34.543434 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:50:34.543424 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls podName:ea5057ea-9ded-470c-8992-4001cbfd7d15 nodeName:}" failed. No retries permitted until 2026-07-17 18:50:35.043402442 +0000 UTC m=+2445.458140452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls") pod "maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" (UID: "ea5057ea-9ded-470c-8992-4001cbfd7d15") : secret "maas-api-e2e-shared-b-1671a0-serving-cert" not found Jul 17 18:50:34.543845 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.543828 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5057ea-9ded-470c-8992-4001cbfd7d15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:34.559604 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:34.559570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25n45\" (UniqueName: \"kubernetes.io/projected/ea5057ea-9ded-470c-8992-4001cbfd7d15-kube-api-access-25n45\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:35.047076 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:35.047046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:35.049572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:35.049542 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:35.229742 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:35.229695 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:35.368767 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:35.368737 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx"] Jul 17 18:50:35.369772 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:50:35.369741 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5057ea_9ded_470c_8992_4001cbfd7d15.slice/crio-6e4e7d818d06c14ee0e485bc413c4644ccdf396e873b609cbc5efcd73f2f5d09 WatchSource:0}: Error finding container 6e4e7d818d06c14ee0e485bc413c4644ccdf396e873b609cbc5efcd73f2f5d09: Status 404 returned error can't find the container with id 6e4e7d818d06c14ee0e485bc413c4644ccdf396e873b609cbc5efcd73f2f5d09 Jul 17 18:50:36.210432 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:36.210390 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" event={"ID":"ea5057ea-9ded-470c-8992-4001cbfd7d15","Type":"ContainerStarted","Data":"30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6"} Jul 17 18:50:36.210618 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:36.210586 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:50:36.210666 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:36.210622 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" event={"ID":"ea5057ea-9ded-470c-8992-4001cbfd7d15","Type":"ContainerStarted","Data":"6e4e7d818d06c14ee0e485bc413c4644ccdf396e873b609cbc5efcd73f2f5d09"} Jul 17 18:50:36.231829 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:36.231778 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" podStartSLOduration=1.544684083 podStartE2EDuration="2.2317616s" totalImagesPullingTime="687.077517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:50:36.231685221 +0000 UTC m=+2446.646423251" watchObservedRunningTime="2026-07-17 18:50:36.2317616 +0000 UTC m=+2446.646499656" Jul 17 18:50:42.224681 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:50:42.224650 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:51:50.391286 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:51:50.391247 2570 scope.go:117] "RemoveContainer" containerID="907ace198095ac975a22f97e3af3373ec468862f4d9cdc8a59e6f941fed7c2c1" Jul 17 18:54:50.254007 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:54:50.253870 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:54:50.260477 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:54:50.260456 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txxzb_03f309d6-aee5-4407-bfc2-a3f9b61cd006/ovn-acl-logging/0.log" Jul 17 18:56:15.546806 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.546720 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7380-140654370164016"] Jul 17 18:56:15.550262 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.550243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:15.552701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.552683 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s686m\"" Jul 17 18:56:15.558007 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.557940 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370164016"] Jul 17 18:56:15.602290 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.602254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c22q2\" (UniqueName: \"kubernetes.io/projected/55060ea7-2e0c-423c-a7cb-8badb183b8f3-kube-api-access-c22q2\") pod \"test-curl-7380-140654370164016\" (UID: \"55060ea7-2e0c-423c-a7cb-8badb183b8f3\") " pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:15.703508 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.703446 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c22q2\" (UniqueName: \"kubernetes.io/projected/55060ea7-2e0c-423c-a7cb-8badb183b8f3-kube-api-access-c22q2\") pod \"test-curl-7380-140654370164016\" (UID: \"55060ea7-2e0c-423c-a7cb-8badb183b8f3\") " pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:15.716079 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.716044 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c22q2\" (UniqueName: \"kubernetes.io/projected/55060ea7-2e0c-423c-a7cb-8badb183b8f3-kube-api-access-c22q2\") pod \"test-curl-7380-140654370164016\" (UID: \"55060ea7-2e0c-423c-a7cb-8badb183b8f3\") " pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:15.861273 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.861177 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:15.997840 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:15.997806 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370164016"] Jul 17 18:56:15.999022 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:56:15.998991 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55060ea7_2e0c_423c_a7cb_8badb183b8f3.slice/crio-187d4169c776f1fa0561821589631aa0553fb2e73c48f4599fcf45f294926258 WatchSource:0}: Error finding container 187d4169c776f1fa0561821589631aa0553fb2e73c48f4599fcf45f294926258: Status 404 returned error can't find the container with id 187d4169c776f1fa0561821589631aa0553fb2e73c48f4599fcf45f294926258 Jul 17 18:56:16.000466 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:16.000444 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:56:16.205170 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:16.205131 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370164016" event={"ID":"55060ea7-2e0c-423c-a7cb-8badb183b8f3","Type":"ContainerStarted","Data":"187d4169c776f1fa0561821589631aa0553fb2e73c48f4599fcf45f294926258"} Jul 17 18:56:17.213641 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:17.213609 2570 generic.go:356] "Generic (PLEG): container finished" podID="55060ea7-2e0c-423c-a7cb-8badb183b8f3" containerID="b59bb360cabdab9c7aaf4e790133a50956bc74a6876d244458dd669223e0e328" exitCode=0 Jul 17 18:56:17.214045 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:17.213700 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370164016" event={"ID":"55060ea7-2e0c-423c-a7cb-8badb183b8f3","Type":"ContainerDied","Data":"b59bb360cabdab9c7aaf4e790133a50956bc74a6876d244458dd669223e0e328"} Jul 17 18:56:18.359115 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.359086 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:18.397254 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.397229 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7380-140654370164016_55060ea7-2e0c-423c-a7cb-8badb183b8f3/test-curl-7380-140654370164016/0.log" Jul 17 18:56:18.422362 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.422319 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7380-140654370164016"] Jul 17 18:56:18.426426 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.426395 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7380-140654370164016"] Jul 17 18:56:18.429982 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.429944 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c22q2\" (UniqueName: \"kubernetes.io/projected/55060ea7-2e0c-423c-a7cb-8badb183b8f3-kube-api-access-c22q2\") pod \"55060ea7-2e0c-423c-a7cb-8badb183b8f3\" (UID: \"55060ea7-2e0c-423c-a7cb-8badb183b8f3\") " Jul 17 18:56:18.432212 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.432184 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55060ea7-2e0c-423c-a7cb-8badb183b8f3-kube-api-access-c22q2" (OuterVolumeSpecName: "kube-api-access-c22q2") pod "55060ea7-2e0c-423c-a7cb-8badb183b8f3" (UID: "55060ea7-2e0c-423c-a7cb-8badb183b8f3"). InnerVolumeSpecName "kube-api-access-c22q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:18.531441 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.531339 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c22q2\" (UniqueName: \"kubernetes.io/projected/55060ea7-2e0c-423c-a7cb-8badb183b8f3-kube-api-access-c22q2\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:18.543551 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.543517 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7380-140654370254896"] Jul 17 18:56:18.543904 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.543892 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55060ea7-2e0c-423c-a7cb-8badb183b8f3" containerName="test-curl-7380-140654370164016" Jul 17 18:56:18.543945 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.543905 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55060ea7-2e0c-423c-a7cb-8badb183b8f3" containerName="test-curl-7380-140654370164016" Jul 17 18:56:18.544010 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.543999 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55060ea7-2e0c-423c-a7cb-8badb183b8f3" containerName="test-curl-7380-140654370164016" Jul 17 18:56:18.547262 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.547242 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370254896" Jul 17 18:56:18.555232 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.555204 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370254896"] Jul 17 18:56:18.632131 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.632094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n978\" (UniqueName: \"kubernetes.io/projected/1fb23edb-3f90-4bfe-98ed-3a219db3d31b-kube-api-access-2n978\") pod \"test-curl-7380-140654370254896\" (UID: \"1fb23edb-3f90-4bfe-98ed-3a219db3d31b\") " pod="opendatahub/test-curl-7380-140654370254896" Jul 17 18:56:18.732739 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.732697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n978\" (UniqueName: \"kubernetes.io/projected/1fb23edb-3f90-4bfe-98ed-3a219db3d31b-kube-api-access-2n978\") pod \"test-curl-7380-140654370254896\" (UID: \"1fb23edb-3f90-4bfe-98ed-3a219db3d31b\") " pod="opendatahub/test-curl-7380-140654370254896" Jul 17 18:56:18.740889 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.740861 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n978\" (UniqueName: \"kubernetes.io/projected/1fb23edb-3f90-4bfe-98ed-3a219db3d31b-kube-api-access-2n978\") pod \"test-curl-7380-140654370254896\" (UID: \"1fb23edb-3f90-4bfe-98ed-3a219db3d31b\") " pod="opendatahub/test-curl-7380-140654370254896" Jul 17 18:56:18.858224 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:18.858126 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370254896" Jul 17 18:56:19.001000 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:19.000975 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370254896"] Jul 17 18:56:19.002576 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:56:19.002539 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb23edb_3f90_4bfe_98ed_3a219db3d31b.slice/crio-4a2ab581e62c984600300587f562be25707555e8a604a45c7e4768d2cf58f309 WatchSource:0}: Error finding container 4a2ab581e62c984600300587f562be25707555e8a604a45c7e4768d2cf58f309: Status 404 returned error can't find the container with id 4a2ab581e62c984600300587f562be25707555e8a604a45c7e4768d2cf58f309 Jul 17 18:56:19.227418 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:19.227393 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:19.227622 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:19.227469 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187d4169c776f1fa0561821589631aa0553fb2e73c48f4599fcf45f294926258" Jul 17 18:56:19.228714 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:19.228680 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370254896" event={"ID":"1fb23edb-3f90-4bfe-98ed-3a219db3d31b","Type":"ContainerStarted","Data":"4a2ab581e62c984600300587f562be25707555e8a604a45c7e4768d2cf58f309"} Jul 17 18:56:19.238341 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:56:19.238309 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7380-140654370164016\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="55060ea7-2e0c-423c-a7cb-8badb183b8f3" pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:20.072116 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:56:20.072074 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7380-140654370164016\" is forbidden: User \"system:node:ip-10-0-137-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-221.ec2.internal' and this object" podUID="55060ea7-2e0c-423c-a7cb-8badb183b8f3" pod="opendatahub/test-curl-7380-140654370164016" Jul 17 18:56:20.073049 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:20.073025 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55060ea7-2e0c-423c-a7cb-8badb183b8f3" path="/var/lib/kubelet/pods/55060ea7-2e0c-423c-a7cb-8badb183b8f3/volumes" Jul 17 18:56:20.236753 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:20.236714 2570 generic.go:356] "Generic (PLEG): container finished" podID="1fb23edb-3f90-4bfe-98ed-3a219db3d31b" containerID="78dfa92d4eb55585daadc2121357271e425513d373a91c478eccb47b650434e1" exitCode=0 Jul 17 18:56:20.236923 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:20.236805 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370254896" event={"ID":"1fb23edb-3f90-4bfe-98ed-3a219db3d31b","Type":"ContainerDied","Data":"78dfa92d4eb55585daadc2121357271e425513d373a91c478eccb47b650434e1"} Jul 17 18:56:21.409933 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.409909 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370254896" Jul 17 18:56:21.430621 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.430581 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7380-140654370254896_1fb23edb-3f90-4bfe-98ed-3a219db3d31b/test-curl-7380-140654370254896/0.log" Jul 17 18:56:21.455445 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.455412 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7380-140654370254896"] Jul 17 18:56:21.459036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.459008 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7380-140654370254896"] Jul 17 18:56:21.559663 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.559568 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n978\" (UniqueName: \"kubernetes.io/projected/1fb23edb-3f90-4bfe-98ed-3a219db3d31b-kube-api-access-2n978\") pod \"1fb23edb-3f90-4bfe-98ed-3a219db3d31b\" (UID: \"1fb23edb-3f90-4bfe-98ed-3a219db3d31b\") " Jul 17 18:56:21.561689 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.561667 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb23edb-3f90-4bfe-98ed-3a219db3d31b-kube-api-access-2n978" (OuterVolumeSpecName: "kube-api-access-2n978") pod "1fb23edb-3f90-4bfe-98ed-3a219db3d31b" (UID: "1fb23edb-3f90-4bfe-98ed-3a219db3d31b"). InnerVolumeSpecName "kube-api-access-2n978". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:21.617101 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.617062 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:21.617443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.617429 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb23edb-3f90-4bfe-98ed-3a219db3d31b" containerName="test-curl-7380-140654370254896" Jul 17 18:56:21.617443 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.617444 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb23edb-3f90-4bfe-98ed-3a219db3d31b" containerName="test-curl-7380-140654370254896" Jul 17 18:56:21.617554 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.617530 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb23edb-3f90-4bfe-98ed-3a219db3d31b" containerName="test-curl-7380-140654370254896" Jul 17 18:56:21.620554 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.620530 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:21.627539 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.627510 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:21.660808 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.660774 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n978\" (UniqueName: \"kubernetes.io/projected/1fb23edb-3f90-4bfe-98ed-3a219db3d31b-kube-api-access-2n978\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:21.762191 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.762157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfksx\" (UniqueName: \"kubernetes.io/projected/6265587b-845d-4cea-beb9-b81ad84c2123-kube-api-access-gfksx\") pod \"test-curl-7380-140654370088256\" (UID: \"6265587b-845d-4cea-beb9-b81ad84c2123\") " pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:21.863806 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.863702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfksx\" (UniqueName: \"kubernetes.io/projected/6265587b-845d-4cea-beb9-b81ad84c2123-kube-api-access-gfksx\") pod \"test-curl-7380-140654370088256\" (UID: \"6265587b-845d-4cea-beb9-b81ad84c2123\") " pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:21.873244 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.873210 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfksx\" (UniqueName: \"kubernetes.io/projected/6265587b-845d-4cea-beb9-b81ad84c2123-kube-api-access-gfksx\") pod \"test-curl-7380-140654370088256\" (UID: \"6265587b-845d-4cea-beb9-b81ad84c2123\") " pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:21.932166 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:21.932127 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:22.071355 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:56:22.071325 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6265587b_845d_4cea_beb9_b81ad84c2123.slice/crio-411a67326742fe9a0557ed6ca3b04f3e9b36334e2849e2eac5bb51b98c2931e5 WatchSource:0}: Error finding container 411a67326742fe9a0557ed6ca3b04f3e9b36334e2849e2eac5bb51b98c2931e5: Status 404 returned error can't find the container with id 411a67326742fe9a0557ed6ca3b04f3e9b36334e2849e2eac5bb51b98c2931e5 Jul 17 18:56:22.073766 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:22.073740 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb23edb-3f90-4bfe-98ed-3a219db3d31b" path="/var/lib/kubelet/pods/1fb23edb-3f90-4bfe-98ed-3a219db3d31b/volumes" Jul 17 18:56:22.074131 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:22.074117 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:22.250331 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:22.250288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370088256" event={"ID":"6265587b-845d-4cea-beb9-b81ad84c2123","Type":"ContainerStarted","Data":"411a67326742fe9a0557ed6ca3b04f3e9b36334e2849e2eac5bb51b98c2931e5"} Jul 17 18:56:22.251633 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:22.251610 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370254896" Jul 17 18:56:22.251769 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:22.251652 2570 scope.go:117] "RemoveContainer" containerID="78dfa92d4eb55585daadc2121357271e425513d373a91c478eccb47b650434e1" Jul 17 18:56:23.260278 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:23.260243 2570 generic.go:356] "Generic (PLEG): container finished" podID="6265587b-845d-4cea-beb9-b81ad84c2123" containerID="f95872b53f24fb3c1e7789af73cce9b736a6a95c464c203529ce259bec3f2352" exitCode=0 Jul 17 18:56:23.260701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:23.260326 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370088256" event={"ID":"6265587b-845d-4cea-beb9-b81ad84c2123","Type":"ContainerDied","Data":"f95872b53f24fb3c1e7789af73cce9b736a6a95c464c203529ce259bec3f2352"} Jul 17 18:56:24.403701 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.403677 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:24.425014 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.424983 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7380-140654370088256_6265587b-845d-4cea-beb9-b81ad84c2123/test-curl-7380-140654370088256/0.log" Jul 17 18:56:24.449745 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.449711 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:24.453539 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.453511 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:24.486912 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.486877 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfksx\" (UniqueName: \"kubernetes.io/projected/6265587b-845d-4cea-beb9-b81ad84c2123-kube-api-access-gfksx\") pod \"6265587b-845d-4cea-beb9-b81ad84c2123\" (UID: \"6265587b-845d-4cea-beb9-b81ad84c2123\") " Jul 17 18:56:24.489016 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.488981 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6265587b-845d-4cea-beb9-b81ad84c2123-kube-api-access-gfksx" (OuterVolumeSpecName: "kube-api-access-gfksx") pod "6265587b-845d-4cea-beb9-b81ad84c2123" (UID: "6265587b-845d-4cea-beb9-b81ad84c2123"). InnerVolumeSpecName "kube-api-access-gfksx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:24.583086 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.583001 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:24.583389 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.583376 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6265587b-845d-4cea-beb9-b81ad84c2123" containerName="test-curl-7380-140654370088256" Jul 17 18:56:24.583438 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.583391 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6265587b-845d-4cea-beb9-b81ad84c2123" containerName="test-curl-7380-140654370088256" Jul 17 18:56:24.583475 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.583462 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6265587b-845d-4cea-beb9-b81ad84c2123" containerName="test-curl-7380-140654370088256" Jul 17 18:56:24.586421 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.586407 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:24.587582 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.587560 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfksx\" (UniqueName: \"kubernetes.io/projected/6265587b-845d-4cea-beb9-b81ad84c2123-kube-api-access-gfksx\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:24.594705 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.594679 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:24.688492 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.688458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lr7\" (UniqueName: \"kubernetes.io/projected/1448cd7c-501c-4ba2-9ea5-2a9074ef1bba-kube-api-access-b8lr7\") pod \"test-curl-7380-140654370088256\" (UID: \"1448cd7c-501c-4ba2-9ea5-2a9074ef1bba\") " pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:24.789420 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.789359 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lr7\" (UniqueName: \"kubernetes.io/projected/1448cd7c-501c-4ba2-9ea5-2a9074ef1bba-kube-api-access-b8lr7\") pod \"test-curl-7380-140654370088256\" (UID: \"1448cd7c-501c-4ba2-9ea5-2a9074ef1bba\") " pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:24.797459 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.797424 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lr7\" (UniqueName: \"kubernetes.io/projected/1448cd7c-501c-4ba2-9ea5-2a9074ef1bba-kube-api-access-b8lr7\") pod \"test-curl-7380-140654370088256\" (UID: \"1448cd7c-501c-4ba2-9ea5-2a9074ef1bba\") " pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:24.896809 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:24.896712 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:25.044261 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:25.044233 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:25.045180 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:56:25.045151 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1448cd7c_501c_4ba2_9ea5_2a9074ef1bba.slice/crio-a52e1ce5fa0cd873824e78b390c4dc5469eaf64a32ec662055b39a789d3dcea0 WatchSource:0}: Error finding container a52e1ce5fa0cd873824e78b390c4dc5469eaf64a32ec662055b39a789d3dcea0: Status 404 returned error can't find the container with id a52e1ce5fa0cd873824e78b390c4dc5469eaf64a32ec662055b39a789d3dcea0 Jul 17 18:56:25.274161 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:25.274125 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:25.274161 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:25.274164 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411a67326742fe9a0557ed6ca3b04f3e9b36334e2849e2eac5bb51b98c2931e5" Jul 17 18:56:25.275517 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:25.275479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370088256" event={"ID":"1448cd7c-501c-4ba2-9ea5-2a9074ef1bba","Type":"ContainerStarted","Data":"a52e1ce5fa0cd873824e78b390c4dc5469eaf64a32ec662055b39a789d3dcea0"} Jul 17 18:56:25.286006 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:25.285972 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7380-140654370088256" oldPodUID="6265587b-845d-4cea-beb9-b81ad84c2123" podUID="1448cd7c-501c-4ba2-9ea5-2a9074ef1bba" Jul 17 18:56:26.074233 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:26.074198 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6265587b-845d-4cea-beb9-b81ad84c2123" path="/var/lib/kubelet/pods/6265587b-845d-4cea-beb9-b81ad84c2123/volumes" Jul 17 18:56:26.282631 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:26.282596 2570 generic.go:356] "Generic (PLEG): container finished" podID="1448cd7c-501c-4ba2-9ea5-2a9074ef1bba" containerID="47ab485d9ececad44c4e1b9699fbf6ee3d04b628f5076913eeb750b9e69b4a23" exitCode=0 Jul 17 18:56:26.282837 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:26.282687 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370088256" event={"ID":"1448cd7c-501c-4ba2-9ea5-2a9074ef1bba","Type":"ContainerDied","Data":"47ab485d9ececad44c4e1b9699fbf6ee3d04b628f5076913eeb750b9e69b4a23"} Jul 17 18:56:27.431872 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.431848 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:27.456063 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.456034 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7380-140654370088256_1448cd7c-501c-4ba2-9ea5-2a9074ef1bba/test-curl-7380-140654370088256/0.log" Jul 17 18:56:27.484350 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.484316 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:27.486604 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.486570 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7380-140654370088256"] Jul 17 18:56:27.513077 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.513038 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8lr7\" (UniqueName: \"kubernetes.io/projected/1448cd7c-501c-4ba2-9ea5-2a9074ef1bba-kube-api-access-b8lr7\") pod \"1448cd7c-501c-4ba2-9ea5-2a9074ef1bba\" (UID: \"1448cd7c-501c-4ba2-9ea5-2a9074ef1bba\") " Jul 17 18:56:27.515198 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.515169 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1448cd7c-501c-4ba2-9ea5-2a9074ef1bba-kube-api-access-b8lr7" (OuterVolumeSpecName: "kube-api-access-b8lr7") pod "1448cd7c-501c-4ba2-9ea5-2a9074ef1bba" (UID: "1448cd7c-501c-4ba2-9ea5-2a9074ef1bba"). InnerVolumeSpecName "kube-api-access-b8lr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:27.601431 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.601337 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:27.601744 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.601730 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1448cd7c-501c-4ba2-9ea5-2a9074ef1bba" containerName="test-curl-7380-140654370088256" Jul 17 18:56:27.601787 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.601745 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1448cd7c-501c-4ba2-9ea5-2a9074ef1bba" containerName="test-curl-7380-140654370088256" Jul 17 18:56:27.601825 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.601808 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1448cd7c-501c-4ba2-9ea5-2a9074ef1bba" containerName="test-curl-7380-140654370088256" Jul 17 18:56:27.604854 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.604837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:27.613155 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.613129 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:27.614007 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.613982 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8lr7\" (UniqueName: \"kubernetes.io/projected/1448cd7c-501c-4ba2-9ea5-2a9074ef1bba-kube-api-access-b8lr7\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:27.714664 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.714618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8z8\" (UniqueName: \"kubernetes.io/projected/826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1-kube-api-access-fq8z8\") pod \"test-curl-7380-140654370089264\" (UID: \"826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1\") " pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:27.815268 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.815231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8z8\" (UniqueName: \"kubernetes.io/projected/826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1-kube-api-access-fq8z8\") pod \"test-curl-7380-140654370089264\" (UID: \"826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1\") " pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:27.824339 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.824307 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8z8\" (UniqueName: \"kubernetes.io/projected/826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1-kube-api-access-fq8z8\") pod \"test-curl-7380-140654370089264\" (UID: \"826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1\") " pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:27.915453 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:27.915356 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:28.055398 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:28.055372 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:28.056466 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:56:28.056439 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod826a25ce_0f6f_4165_bd2f_96b7bc9fe5f1.slice/crio-bd818fe8a6297fdeead5838de7b0a22cdbe5420c23e93a23b88bc8bc74d44b89 WatchSource:0}: Error finding container bd818fe8a6297fdeead5838de7b0a22cdbe5420c23e93a23b88bc8bc74d44b89: Status 404 returned error can't find the container with id bd818fe8a6297fdeead5838de7b0a22cdbe5420c23e93a23b88bc8bc74d44b89 Jul 17 18:56:28.072970 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:28.072928 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1448cd7c-501c-4ba2-9ea5-2a9074ef1bba" path="/var/lib/kubelet/pods/1448cd7c-501c-4ba2-9ea5-2a9074ef1bba/volumes" Jul 17 18:56:28.296342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:28.296309 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370089264" event={"ID":"826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1","Type":"ContainerStarted","Data":"bd818fe8a6297fdeead5838de7b0a22cdbe5420c23e93a23b88bc8bc74d44b89"} Jul 17 18:56:28.297563 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:28.297538 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370088256" Jul 17 18:56:28.297695 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:28.297572 2570 scope.go:117] "RemoveContainer" containerID="47ab485d9ececad44c4e1b9699fbf6ee3d04b628f5076913eeb750b9e69b4a23" Jul 17 18:56:29.306320 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:29.306281 2570 generic.go:356] "Generic (PLEG): container finished" podID="826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1" containerID="a5a84ef2a4bcbe63c3b55de95482cb187cde8c1fd9a55ce7345c2f1768352ef7" exitCode=0 Jul 17 18:56:29.306778 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:29.306365 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370089264" event={"ID":"826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1","Type":"ContainerDied","Data":"a5a84ef2a4bcbe63c3b55de95482cb187cde8c1fd9a55ce7345c2f1768352ef7"} Jul 17 18:56:30.457644 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.457617 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:30.480304 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.480270 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7380-140654370089264_826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1/test-curl-7380-140654370089264/0.log" Jul 17 18:56:30.506389 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.506357 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:30.508994 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.508970 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:30.634852 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.634775 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:30.635150 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.635136 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1" containerName="test-curl-7380-140654370089264" Jul 17 18:56:30.635194 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.635152 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1" containerName="test-curl-7380-140654370089264" Jul 17 18:56:30.635234 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.635222 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1" containerName="test-curl-7380-140654370089264" Jul 17 18:56:30.638203 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.638182 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq8z8\" (UniqueName: \"kubernetes.io/projected/826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1-kube-api-access-fq8z8\") pod \"826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1\" (UID: \"826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1\") " Jul 17 18:56:30.638295 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.638187 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:30.640284 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.640259 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1-kube-api-access-fq8z8" (OuterVolumeSpecName: "kube-api-access-fq8z8") pod "826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1" (UID: "826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1"). InnerVolumeSpecName "kube-api-access-fq8z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:30.652196 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.652170 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:30.738813 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.738770 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxd9\" (UniqueName: \"kubernetes.io/projected/6fb213b9-9ecf-4232-8c18-080b3ed3f4d8-kube-api-access-srxd9\") pod \"test-curl-7380-140654370089264\" (UID: \"6fb213b9-9ecf-4232-8c18-080b3ed3f4d8\") " pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:30.739036 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.738886 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8z8\" (UniqueName: \"kubernetes.io/projected/826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1-kube-api-access-fq8z8\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:30.840072 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.840040 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxd9\" (UniqueName: \"kubernetes.io/projected/6fb213b9-9ecf-4232-8c18-080b3ed3f4d8-kube-api-access-srxd9\") pod \"test-curl-7380-140654370089264\" (UID: \"6fb213b9-9ecf-4232-8c18-080b3ed3f4d8\") " pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:30.848793 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.848761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxd9\" (UniqueName: \"kubernetes.io/projected/6fb213b9-9ecf-4232-8c18-080b3ed3f4d8-kube-api-access-srxd9\") pod \"test-curl-7380-140654370089264\" (UID: \"6fb213b9-9ecf-4232-8c18-080b3ed3f4d8\") " pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:30.954885 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:30.954850 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:31.091740 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:31.090626 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:31.094589 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:56:31.094561 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fb213b9_9ecf_4232_8c18_080b3ed3f4d8.slice/crio-90081b4766b3325b45159cbafda7f9940d8957e6b763151a3618d04e7ca6a918 WatchSource:0}: Error finding container 90081b4766b3325b45159cbafda7f9940d8957e6b763151a3618d04e7ca6a918: Status 404 returned error can't find the container with id 90081b4766b3325b45159cbafda7f9940d8957e6b763151a3618d04e7ca6a918 Jul 17 18:56:31.321766 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:31.321683 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:31.321916 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:31.321681 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd818fe8a6297fdeead5838de7b0a22cdbe5420c23e93a23b88bc8bc74d44b89" Jul 17 18:56:31.323102 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:31.323074 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370089264" event={"ID":"6fb213b9-9ecf-4232-8c18-080b3ed3f4d8","Type":"ContainerStarted","Data":"90081b4766b3325b45159cbafda7f9940d8957e6b763151a3618d04e7ca6a918"} Jul 17 18:56:31.334370 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:31.334343 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7380-140654370089264" oldPodUID="826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1" podUID="6fb213b9-9ecf-4232-8c18-080b3ed3f4d8" Jul 17 18:56:32.072385 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:32.072349 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1" path="/var/lib/kubelet/pods/826a25ce-0f6f-4165-bd2f-96b7bc9fe5f1/volumes" Jul 17 18:56:32.330650 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:32.330546 2570 generic.go:356] "Generic (PLEG): container finished" podID="6fb213b9-9ecf-4232-8c18-080b3ed3f4d8" containerID="2937052bc4be741a6a6d7d858fe405f4c949d1409283fc42e52d2b656f3d8f4a" exitCode=0 Jul 17 18:56:32.330650 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:32.330641 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7380-140654370089264" event={"ID":"6fb213b9-9ecf-4232-8c18-080b3ed3f4d8","Type":"ContainerDied","Data":"2937052bc4be741a6a6d7d858fe405f4c949d1409283fc42e52d2b656f3d8f4a"} Jul 17 18:56:33.474143 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.474121 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:33.494663 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.494639 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7380-140654370089264_6fb213b9-9ecf-4232-8c18-080b3ed3f4d8/test-curl-7380-140654370089264/0.log" Jul 17 18:56:33.518556 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.518515 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:33.522436 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.522410 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7380-140654370089264"] Jul 17 18:56:33.563897 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.563863 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srxd9\" (UniqueName: \"kubernetes.io/projected/6fb213b9-9ecf-4232-8c18-080b3ed3f4d8-kube-api-access-srxd9\") pod \"6fb213b9-9ecf-4232-8c18-080b3ed3f4d8\" (UID: \"6fb213b9-9ecf-4232-8c18-080b3ed3f4d8\") " Jul 17 18:56:33.565875 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.565850 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb213b9-9ecf-4232-8c18-080b3ed3f4d8-kube-api-access-srxd9" (OuterVolumeSpecName: "kube-api-access-srxd9") pod "6fb213b9-9ecf-4232-8c18-080b3ed3f4d8" (UID: "6fb213b9-9ecf-4232-8c18-080b3ed3f4d8"). InnerVolumeSpecName "kube-api-access-srxd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:33.664651 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.664563 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srxd9\" (UniqueName: \"kubernetes.io/projected/6fb213b9-9ecf-4232-8c18-080b3ed3f4d8-kube-api-access-srxd9\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:33.711470 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.711436 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c"] Jul 17 18:56:33.711819 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.711806 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb213b9-9ecf-4232-8c18-080b3ed3f4d8" containerName="test-curl-7380-140654370089264" Jul 17 18:56:33.711859 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.711821 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb213b9-9ecf-4232-8c18-080b3ed3f4d8" containerName="test-curl-7380-140654370089264" Jul 17 18:56:33.711899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.711884 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fb213b9-9ecf-4232-8c18-080b3ed3f4d8" containerName="test-curl-7380-140654370089264" Jul 17 18:56:33.714871 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.714854 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" Jul 17 18:56:33.717453 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.717431 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:56:33.724671 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.724640 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c"] Jul 17 18:56:33.866889 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.866850 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f9f2e-d684-4699-9bd2-f7dc0931304c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c\" (UID: \"6e1f9f2e-d684-4699-9bd2-f7dc0931304c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" Jul 17 18:56:33.968059 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.968028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f9f2e-d684-4699-9bd2-f7dc0931304c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c\" (UID: \"6e1f9f2e-d684-4699-9bd2-f7dc0931304c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" Jul 17 18:56:33.968671 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:33.968639 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f9f2e-d684-4699-9bd2-f7dc0931304c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c\" (UID: \"6e1f9f2e-d684-4699-9bd2-f7dc0931304c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" Jul 17 18:56:34.026533 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:34.026494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" Jul 17 18:56:34.074632 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:34.074601 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb213b9-9ecf-4232-8c18-080b3ed3f4d8" path="/var/lib/kubelet/pods/6fb213b9-9ecf-4232-8c18-080b3ed3f4d8/volumes" Jul 17 18:56:34.166228 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:34.166199 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c"] Jul 17 18:56:34.166862 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:56:34.166837 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1f9f2e_d684_4699_9bd2_f7dc0931304c.slice/crio-28a1fe9185100a28708f8b292258fffb87c83ddb7b4ee04308f65a333c3a48c5 WatchSource:0}: Error finding container 28a1fe9185100a28708f8b292258fffb87c83ddb7b4ee04308f65a333c3a48c5: Status 404 returned error can't find the container with id 28a1fe9185100a28708f8b292258fffb87c83ddb7b4ee04308f65a333c3a48c5 Jul 17 18:56:34.345141 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:34.345109 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7380-140654370089264" Jul 17 18:56:34.345274 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:34.345144 2570 scope.go:117] "RemoveContainer" containerID="2937052bc4be741a6a6d7d858fe405f4c949d1409283fc42e52d2b656f3d8f4a" Jul 17 18:56:34.346826 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:34.346793 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" event={"ID":"6e1f9f2e-d684-4699-9bd2-f7dc0931304c","Type":"ContainerStarted","Data":"28a1fe9185100a28708f8b292258fffb87c83ddb7b4ee04308f65a333c3a48c5"} Jul 17 18:56:35.355287 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:35.355252 2570 generic.go:356] "Generic (PLEG): container finished" podID="6e1f9f2e-d684-4699-9bd2-f7dc0931304c" containerID="daf29c3551d164a136b09106c6e60e42d47972c0ae03049ca4f3ffb2ee47db99" exitCode=0 Jul 17 18:56:35.355685 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:35.355342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" event={"ID":"6e1f9f2e-d684-4699-9bd2-f7dc0931304c","Type":"ContainerDied","Data":"daf29c3551d164a136b09106c6e60e42d47972c0ae03049ca4f3ffb2ee47db99"} Jul 17 18:56:36.496066 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:36.496036 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" Jul 17 18:56:36.691324 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:36.691292 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f9f2e-d684-4699-9bd2-f7dc0931304c-maas-api-service-ca\") pod \"6e1f9f2e-d684-4699-9bd2-f7dc0931304c\" (UID: \"6e1f9f2e-d684-4699-9bd2-f7dc0931304c\") " Jul 17 18:56:36.691690 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:36.691663 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1f9f2e-d684-4699-9bd2-f7dc0931304c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e1f9f2e-d684-4699-9bd2-f7dc0931304c" (UID: "6e1f9f2e-d684-4699-9bd2-f7dc0931304c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:36.792349 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:36.792298 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f9f2e-d684-4699-9bd2-f7dc0931304c-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:37.367717 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:37.367687 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" Jul 17 18:56:37.367717 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:37.367692 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c" event={"ID":"6e1f9f2e-d684-4699-9bd2-f7dc0931304c","Type":"ContainerDied","Data":"28a1fe9185100a28708f8b292258fffb87c83ddb7b4ee04308f65a333c3a48c5"} Jul 17 18:56:37.368019 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:37.367747 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28a1fe9185100a28708f8b292258fffb87c83ddb7b4ee04308f65a333c3a48c5" Jul 17 18:56:44.218853 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.218813 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7"] Jul 17 18:56:44.219327 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.219026 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" podUID="2140c468-aab8-4959-93ff-1a78d1848991" containerName="maas-api" containerID="cri-o://4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382" gracePeriod=30 Jul 17 18:56:44.665018 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.664971 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:56:44.755203 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.755115 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/2140c468-aab8-4959-93ff-1a78d1848991-kube-api-access-dpxq4\") pod \"2140c468-aab8-4959-93ff-1a78d1848991\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " Jul 17 18:56:44.755203 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.755170 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls\") pod \"2140c468-aab8-4959-93ff-1a78d1848991\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " Jul 17 18:56:44.755409 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.755214 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140c468-aab8-4959-93ff-1a78d1848991-openshift-service-ca\") pod \"2140c468-aab8-4959-93ff-1a78d1848991\" (UID: \"2140c468-aab8-4959-93ff-1a78d1848991\") " Jul 17 18:56:44.755632 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.755600 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2140c468-aab8-4959-93ff-1a78d1848991-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2140c468-aab8-4959-93ff-1a78d1848991" (UID: "2140c468-aab8-4959-93ff-1a78d1848991"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:44.757339 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.757314 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2140c468-aab8-4959-93ff-1a78d1848991-kube-api-access-dpxq4" (OuterVolumeSpecName: "kube-api-access-dpxq4") pod "2140c468-aab8-4959-93ff-1a78d1848991" (UID: "2140c468-aab8-4959-93ff-1a78d1848991"). InnerVolumeSpecName "kube-api-access-dpxq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:44.757454 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.757434 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2140c468-aab8-4959-93ff-1a78d1848991" (UID: "2140c468-aab8-4959-93ff-1a78d1848991"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:56:44.855898 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.855859 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/2140c468-aab8-4959-93ff-1a78d1848991-kube-api-access-dpxq4\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:44.855898 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.855892 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2140c468-aab8-4959-93ff-1a78d1848991-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:44.855898 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:44.855907 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2140c468-aab8-4959-93ff-1a78d1848991-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:56:45.417270 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.417238 2570 generic.go:356] "Generic (PLEG): container finished" podID="2140c468-aab8-4959-93ff-1a78d1848991" containerID="4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382" exitCode=0 Jul 17 18:56:45.417783 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.417291 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" Jul 17 18:56:45.417783 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.417327 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" event={"ID":"2140c468-aab8-4959-93ff-1a78d1848991","Type":"ContainerDied","Data":"4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382"} Jul 17 18:56:45.417783 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.417384 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7" event={"ID":"2140c468-aab8-4959-93ff-1a78d1848991","Type":"ContainerDied","Data":"0235fb64c4a726a46c9002e94bfe711ffaf227343ffcd9ae2427ebbc8d825dbe"} Jul 17 18:56:45.417783 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.417413 2570 scope.go:117] "RemoveContainer" containerID="4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382" Jul 17 18:56:45.427879 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.427853 2570 scope.go:117] "RemoveContainer" containerID="4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382" Jul 17 18:56:45.428175 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:56:45.428156 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382\": container with ID starting with 4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382 not found: ID does not exist" containerID="4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382" Jul 17 18:56:45.428248 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.428187 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382"} err="failed to get container status \"4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382\": rpc error: code = NotFound desc = could not find container \"4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382\": container with ID starting with 4b9927087ac28134dc91207e53859ad81dd18ec9ae676a7ff2b72eb187ddf382 not found: ID does not exist" Jul 17 18:56:45.440827 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.440794 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7"] Jul 17 18:56:45.444751 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:45.444718 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4280a6-6fbd4c9f58-jf4c7"] Jul 17 18:56:46.072487 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:46.072456 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2140c468-aab8-4959-93ff-1a78d1848991" path="/var/lib/kubelet/pods/2140c468-aab8-4959-93ff-1a78d1848991/volumes" Jul 17 18:56:59.879855 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.879821 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml"] Jul 17 18:56:59.880342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.880240 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1f9f2e-d684-4699-9bd2-f7dc0931304c" containerName="revoke-keys" Jul 17 18:56:59.880342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.880253 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1f9f2e-d684-4699-9bd2-f7dc0931304c" containerName="revoke-keys" Jul 17 18:56:59.880342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.880268 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2140c468-aab8-4959-93ff-1a78d1848991" containerName="maas-api" Jul 17 18:56:59.880342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.880274 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2140c468-aab8-4959-93ff-1a78d1848991" containerName="maas-api" Jul 17 18:56:59.880481 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.880349 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1f9f2e-d684-4699-9bd2-f7dc0931304c" containerName="revoke-keys" Jul 17 18:56:59.880481 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.880359 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2140c468-aab8-4959-93ff-1a78d1848991" containerName="maas-api" Jul 17 18:56:59.883453 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.883433 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" Jul 17 18:56:59.886126 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.886106 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g94w2\"" Jul 17 18:56:59.892641 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.892617 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml"] Jul 17 18:56:59.986099 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:56:59.986059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c3591-686a-4b18-9365-f4573efed60a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1671a0-44mml\" (UID: \"421c3591-686a-4b18-9365-f4573efed60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" Jul 17 18:57:00.087204 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:00.087167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c3591-686a-4b18-9365-f4573efed60a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1671a0-44mml\" (UID: \"421c3591-686a-4b18-9365-f4573efed60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" Jul 17 18:57:00.087814 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:00.087788 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c3591-686a-4b18-9365-f4573efed60a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1671a0-44mml\" (UID: \"421c3591-686a-4b18-9365-f4573efed60a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" Jul 17 18:57:00.195509 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:00.195470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" Jul 17 18:57:00.336180 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:00.336146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml"] Jul 17 18:57:00.336526 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:57:00.336504 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421c3591_686a_4b18_9365_f4573efed60a.slice/crio-cd99b5dd0c5f5dc5e01a7097ec5943fc1783075674277a467cfcb7883410f3d6 WatchSource:0}: Error finding container cd99b5dd0c5f5dc5e01a7097ec5943fc1783075674277a467cfcb7883410f3d6: Status 404 returned error can't find the container with id cd99b5dd0c5f5dc5e01a7097ec5943fc1783075674277a467cfcb7883410f3d6 Jul 17 18:57:00.514050 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:00.513921 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" event={"ID":"421c3591-686a-4b18-9365-f4573efed60a","Type":"ContainerStarted","Data":"cd99b5dd0c5f5dc5e01a7097ec5943fc1783075674277a467cfcb7883410f3d6"} Jul 17 18:57:01.522449 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:01.522407 2570 generic.go:356] "Generic (PLEG): container finished" podID="421c3591-686a-4b18-9365-f4573efed60a" containerID="8b1cfc5649e97414d23ab9559069654af4da7e607de05faa17710d18a78616fc" exitCode=0 Jul 17 18:57:01.522831 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:01.522488 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" event={"ID":"421c3591-686a-4b18-9365-f4573efed60a","Type":"ContainerDied","Data":"8b1cfc5649e97414d23ab9559069654af4da7e607de05faa17710d18a78616fc"} Jul 17 18:57:02.664969 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:02.664937 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" Jul 17 18:57:02.711658 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:02.711621 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c3591-686a-4b18-9365-f4573efed60a-maas-api-service-ca\") pod \"421c3591-686a-4b18-9365-f4573efed60a\" (UID: \"421c3591-686a-4b18-9365-f4573efed60a\") " Jul 17 18:57:02.712006 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:02.711985 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421c3591-686a-4b18-9365-f4573efed60a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "421c3591-686a-4b18-9365-f4573efed60a" (UID: "421c3591-686a-4b18-9365-f4573efed60a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:57:02.812600 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:02.812511 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421c3591-686a-4b18-9365-f4573efed60a-maas-api-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:57:03.535891 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:03.535858 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" Jul 17 18:57:03.536117 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:03.535891 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1671a0-44mml" event={"ID":"421c3591-686a-4b18-9365-f4573efed60a","Type":"ContainerDied","Data":"cd99b5dd0c5f5dc5e01a7097ec5943fc1783075674277a467cfcb7883410f3d6"} Jul 17 18:57:03.536117 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:03.535950 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd99b5dd0c5f5dc5e01a7097ec5943fc1783075674277a467cfcb7883410f3d6" Jul 17 18:57:10.386471 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.386014 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx"] Jul 17 18:57:10.386471 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.386266 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" podUID="ea5057ea-9ded-470c-8992-4001cbfd7d15" containerName="maas-api" containerID="cri-o://30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6" gracePeriod=30 Jul 17 18:57:10.840740 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.840709 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:57:10.984191 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.984157 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls\") pod \"ea5057ea-9ded-470c-8992-4001cbfd7d15\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " Jul 17 18:57:10.984191 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.984200 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25n45\" (UniqueName: \"kubernetes.io/projected/ea5057ea-9ded-470c-8992-4001cbfd7d15-kube-api-access-25n45\") pod \"ea5057ea-9ded-470c-8992-4001cbfd7d15\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " Jul 17 18:57:10.984416 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.984248 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5057ea-9ded-470c-8992-4001cbfd7d15-openshift-service-ca\") pod \"ea5057ea-9ded-470c-8992-4001cbfd7d15\" (UID: \"ea5057ea-9ded-470c-8992-4001cbfd7d15\") " Jul 17 18:57:10.984626 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.984599 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5057ea-9ded-470c-8992-4001cbfd7d15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea5057ea-9ded-470c-8992-4001cbfd7d15" (UID: "ea5057ea-9ded-470c-8992-4001cbfd7d15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:57:10.986240 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.986215 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea5057ea-9ded-470c-8992-4001cbfd7d15" (UID: "ea5057ea-9ded-470c-8992-4001cbfd7d15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:57:10.986414 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:10.986393 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5057ea-9ded-470c-8992-4001cbfd7d15-kube-api-access-25n45" (OuterVolumeSpecName: "kube-api-access-25n45") pod "ea5057ea-9ded-470c-8992-4001cbfd7d15" (UID: "ea5057ea-9ded-470c-8992-4001cbfd7d15"). InnerVolumeSpecName "kube-api-access-25n45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:57:11.085279 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.085240 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5057ea-9ded-470c-8992-4001cbfd7d15-openshift-service-ca\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:57:11.085279 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.085271 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5057ea-9ded-470c-8992-4001cbfd7d15-maas-api-tls\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:57:11.085279 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.085281 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25n45\" (UniqueName: \"kubernetes.io/projected/ea5057ea-9ded-470c-8992-4001cbfd7d15-kube-api-access-25n45\") on node \"ip-10-0-137-221.ec2.internal\" DevicePath \"\"" Jul 17 18:57:11.582412 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.582372 2570 generic.go:356] "Generic (PLEG): container finished" podID="ea5057ea-9ded-470c-8992-4001cbfd7d15" containerID="30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6" exitCode=0 Jul 17 18:57:11.582412 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.582415 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" Jul 17 18:57:11.582979 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.582450 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" event={"ID":"ea5057ea-9ded-470c-8992-4001cbfd7d15","Type":"ContainerDied","Data":"30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6"} Jul 17 18:57:11.582979 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.582501 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx" event={"ID":"ea5057ea-9ded-470c-8992-4001cbfd7d15","Type":"ContainerDied","Data":"6e4e7d818d06c14ee0e485bc413c4644ccdf396e873b609cbc5efcd73f2f5d09"} Jul 17 18:57:11.582979 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.582525 2570 scope.go:117] "RemoveContainer" containerID="30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6" Jul 17 18:57:11.592191 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.592173 2570 scope.go:117] "RemoveContainer" containerID="30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6" Jul 17 18:57:11.592461 ip-10-0-137-221 kubenswrapper[2570]: E0717 18:57:11.592443 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6\": container with ID starting with 30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6 not found: ID does not exist" containerID="30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6" Jul 17 18:57:11.592504 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.592471 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6"} err="failed to get container status \"30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6\": rpc error: code = NotFound desc = could not find container \"30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6\": container with ID starting with 30ba5cabc8eda27a4e14ec8132eab18cf6cc2c546561aca0067fe10fbe22f8e6 not found: ID does not exist" Jul 17 18:57:11.604589 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.604561 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx"] Jul 17 18:57:11.608572 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:11.608549 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1671a0-fb57b8cf8-j86cx"] Jul 17 18:57:12.073363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:12.073331 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5057ea-9ded-470c-8992-4001cbfd7d15" path="/var/lib/kubelet/pods/ea5057ea-9ded-470c-8992-4001cbfd7d15/volumes" Jul 17 18:57:26.297575 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:26.297536 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86bd5d5fbc-5tzqt_c22be84f-0691-46e6-a7d1-851775186304/authorino/0.log" Jul 17 18:57:31.677579 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:31.677543 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tj59h_f57471e8-c7c0-4122-861f-a191ed9c595c/manager/0.log" Jul 17 18:57:31.802528 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:31.802502 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rdvx5_ac647307-f36b-449e-83ea-c4ba30e69624/manager/0.log" Jul 17 18:57:32.061881 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:32.061852 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qbzt6_8ab02e79-2ce3-4217-a8ac-567ae14b3f46/server/0.log" Jul 17 18:57:32.193514 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:32.193479 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hvl2r_22297a5e-a9e3-4741-9466-62f8c3f46878/manager/0.log" Jul 17 18:57:32.320334 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:32.320249 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-2qtd6_54850ee9-6cb1-4e0e-9343-15092fc4581a/copy-manifests/0.log" Jul 17 18:57:32.352290 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:32.352266 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-2qtd6_54850ee9-6cb1-4e0e-9343-15092fc4581a/manager/0.log" Jul 17 18:57:33.161549 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:33.161515 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738550-c67w9_9d7f416d-4f3f-40ec-8455-52b68d6b6773/cleanup/0.log" Jul 17 18:57:33.276425 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:33.276396 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738565-lkjh2_77c96b3a-a3e7-419d-abe9-b71d5a4d1ed3/cleanup/0.log" Jul 17 18:57:33.398422 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:33.398385 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-9b0be972-w4r55_27eb318b-ea60-4146-8a3e-aeafbe944bcc/revoke-keys/0.log" Jul 17 18:57:33.537322 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:33.537292 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-a8d2046a-kwr2f_e4442376-3267-46fa-aeca-4289d36167db/revoke-keys/0.log" Jul 17 18:57:33.652977 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:33.652924 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-82b1be-942r8_279ff674-2c68-4073-9e9e-ef186afcac0e/revoke-keys/0.log" Jul 17 18:57:33.774904 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:33.774872 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-2cfeb4-mp7n9_337e7c03-8cb0-4052-b5e0-6dfebac50ba8/revoke-keys/0.log" Jul 17 18:57:33.907707 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:33.907629 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-1b52ad55-ddz8q_727d16f5-baf2-4f1c-a4c4-5792fe8cec92/revoke-keys/0.log" Jul 17 18:57:34.031646 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.031616 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-764e6c-f7h62_426e5e33-c125-4281-aa1a-c970fc804887/revoke-keys/0.log" Jul 17 18:57:34.151107 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.151082 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-8f4385-rgrdh_ab52254d-cd30-47db-86be-3098345ac7fd/revoke-keys/0.log" Jul 17 18:57:34.267981 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.267926 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-d37232e3-kq8qm_54926b45-223b-43ae-85a3-99a1ff06e4d0/revoke-keys/0.log" Jul 17 18:57:34.382969 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.382939 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-5a93a056-j9zdb_4259ecae-7656-41df-9e6b-25ac31ce116d/revoke-keys/0.log" Jul 17 18:57:34.494778 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.494750 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ba643e57-xk2rs_741ebdeb-9476-45d9-81f7-91e499c1462c/revoke-keys/0.log" Jul 17 18:57:34.610965 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.610877 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-4280a6-vpj2c_6e1f9f2e-d684-4699-9bd2-f7dc0931304c/revoke-keys/0.log" Jul 17 18:57:34.729696 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.729666 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-1671a0-44mml_421c3591-686a-4b18-9365-f4573efed60a/revoke-keys/0.log" Jul 17 18:57:34.853450 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:34.853421 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ns9zd_acb353a2-7707-401e-8205-9b50e8706be8/postgres/0.log" Jul 17 18:57:35.633982 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.633924 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs_938899aa-38b1-42d7-a873-df2b0aef42eb/pull/0.log" Jul 17 18:57:35.640363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.640338 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs_938899aa-38b1-42d7-a873-df2b0aef42eb/extract/0.log" Jul 17 18:57:35.646709 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.646691 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs_938899aa-38b1-42d7-a873-df2b0aef42eb/util/0.log" Jul 17 18:57:35.761289 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.761261 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m_36c88212-fb53-414b-87a7-a412c0aab5a1/util/0.log" Jul 17 18:57:35.768053 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.768024 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m_36c88212-fb53-414b-87a7-a412c0aab5a1/pull/0.log" Jul 17 18:57:35.774559 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.774542 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m_36c88212-fb53-414b-87a7-a412c0aab5a1/extract/0.log" Jul 17 18:57:35.910723 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.910638 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n_a6441300-acfd-4e7e-87bc-463e6baccaa3/util/0.log" Jul 17 18:57:35.925677 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.925649 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n_a6441300-acfd-4e7e-87bc-463e6baccaa3/pull/0.log" Jul 17 18:57:35.937931 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:35.937906 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n_a6441300-acfd-4e7e-87bc-463e6baccaa3/extract/0.log" Jul 17 18:57:36.059869 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.059843 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz_273b7e4d-641c-4591-a119-cc72059acf9d/util/0.log" Jul 17 18:57:36.066652 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.066625 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz_273b7e4d-641c-4591-a119-cc72059acf9d/pull/0.log" Jul 17 18:57:36.074479 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.074457 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz_273b7e4d-641c-4591-a119-cc72059acf9d/extract/0.log" Jul 17 18:57:36.196711 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.196687 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86bd5d5fbc-5tzqt_c22be84f-0691-46e6-a7d1-851775186304/authorino/0.log" Jul 17 18:57:36.318246 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.318213 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nt6ns_03f56fa4-6812-4702-aa55-71d3c7814f5e/manager/0.log" Jul 17 18:57:36.431720 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.431695 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ftlmm_bdc997a8-42fe-43da-8c59-40ed30a843b7/manager/0.log" Jul 17 18:57:36.660014 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.659936 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lc78v_cf5e7b80-d007-4d25-a6ea-f59634b988cd/registry-server/0.log" Jul 17 18:57:36.784352 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:36.784311 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-krrgw_cc1e0fff-6947-4365-878e-282091ed545b/manager/0.log" Jul 17 18:57:37.014251 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:37.014223 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-f7lfn_7e5dd04e-25db-4ef2-9694-2e2f0993ffc3/manager/0.log" Jul 17 18:57:37.607614 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:37.607583 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-k8lbs_a2bd523b-e875-4aea-b1f4-def4f0875183/kube-auth-proxy/0.log" Jul 17 18:57:37.964857 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:37.964831 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-4kd6z_663d5575-ebca-4529-a9ef-9bbbd6137773/payload-pre-processing/0.log" Jul 17 18:57:38.553251 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:38.553223 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nc2l9_4953ecfb-aa53-46cf-a313-481b07f060b2/main/0.log" Jul 17 18:57:44.470918 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.470839 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5s9n5/must-gather-c9h9m"] Jul 17 18:57:44.471765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.471222 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421c3591-686a-4b18-9365-f4573efed60a" containerName="revoke-keys" Jul 17 18:57:44.471765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.471234 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="421c3591-686a-4b18-9365-f4573efed60a" containerName="revoke-keys" Jul 17 18:57:44.471765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.471247 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5057ea-9ded-470c-8992-4001cbfd7d15" containerName="maas-api" Jul 17 18:57:44.471765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.471253 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5057ea-9ded-470c-8992-4001cbfd7d15" containerName="maas-api" Jul 17 18:57:44.471765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.471308 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="421c3591-686a-4b18-9365-f4573efed60a" containerName="revoke-keys" Jul 17 18:57:44.471765 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.471319 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea5057ea-9ded-470c-8992-4001cbfd7d15" containerName="maas-api" Jul 17 18:57:44.474497 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.474481 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.477204 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.477181 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s9n5\"/\"openshift-service-ca.crt\"" Jul 17 18:57:44.478197 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.478181 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5s9n5\"/\"default-dockercfg-dw4zq\"" Jul 17 18:57:44.478251 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.478227 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s9n5\"/\"kube-root-ca.crt\"" Jul 17 18:57:44.483328 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.483309 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s9n5/must-gather-c9h9m"] Jul 17 18:57:44.566234 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.566195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ghn\" (UniqueName: \"kubernetes.io/projected/6f901f11-ae1f-4281-8854-f99ca894bb74-kube-api-access-g5ghn\") pod \"must-gather-c9h9m\" (UID: \"6f901f11-ae1f-4281-8854-f99ca894bb74\") " pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.566415 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.566257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6f901f11-ae1f-4281-8854-f99ca894bb74-must-gather-output\") pod \"must-gather-c9h9m\" (UID: \"6f901f11-ae1f-4281-8854-f99ca894bb74\") " pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.667478 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.667434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6f901f11-ae1f-4281-8854-f99ca894bb74-must-gather-output\") pod \"must-gather-c9h9m\" (UID: \"6f901f11-ae1f-4281-8854-f99ca894bb74\") " pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.667691 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.667531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ghn\" (UniqueName: \"kubernetes.io/projected/6f901f11-ae1f-4281-8854-f99ca894bb74-kube-api-access-g5ghn\") pod \"must-gather-c9h9m\" (UID: \"6f901f11-ae1f-4281-8854-f99ca894bb74\") " pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.667794 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.667773 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6f901f11-ae1f-4281-8854-f99ca894bb74-must-gather-output\") pod \"must-gather-c9h9m\" (UID: \"6f901f11-ae1f-4281-8854-f99ca894bb74\") " pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.678790 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.678761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ghn\" (UniqueName: \"kubernetes.io/projected/6f901f11-ae1f-4281-8854-f99ca894bb74-kube-api-access-g5ghn\") pod \"must-gather-c9h9m\" (UID: \"6f901f11-ae1f-4281-8854-f99ca894bb74\") " pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.784342 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.784253 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s9n5/must-gather-c9h9m" Jul 17 18:57:44.916003 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:44.915973 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s9n5/must-gather-c9h9m"] Jul 17 18:57:44.917102 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:57:44.917074 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f901f11_ae1f_4281_8854_f99ca894bb74.slice/crio-b295d5da427eb58b7cf08599f034bac1f5cbcaaec2c86a215d445ec02ab46db7 WatchSource:0}: Error finding container b295d5da427eb58b7cf08599f034bac1f5cbcaaec2c86a215d445ec02ab46db7: Status 404 returned error can't find the container with id b295d5da427eb58b7cf08599f034bac1f5cbcaaec2c86a215d445ec02ab46db7 Jul 17 18:57:45.795212 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:45.795188 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s9n5/must-gather-c9h9m" event={"ID":"6f901f11-ae1f-4281-8854-f99ca894bb74","Type":"ContainerStarted","Data":"b295d5da427eb58b7cf08599f034bac1f5cbcaaec2c86a215d445ec02ab46db7"} Jul 17 18:57:46.806735 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:46.806688 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s9n5/must-gather-c9h9m" event={"ID":"6f901f11-ae1f-4281-8854-f99ca894bb74","Type":"ContainerStarted","Data":"a0d6d16a0e818b0921bd5d1e7f4ffe5b388c1befb8125bd4690080fb9f52adea"} Jul 17 18:57:46.807277 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:46.806749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s9n5/must-gather-c9h9m" event={"ID":"6f901f11-ae1f-4281-8854-f99ca894bb74","Type":"ContainerStarted","Data":"96743af774e3d66a1c752e4c5089c1e12548da47817a6c15b3f45b8ffddf41d6"} Jul 17 18:57:46.828066 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:46.828015 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5s9n5/must-gather-c9h9m" podStartSLOduration=1.960307043 podStartE2EDuration="2.82799893s" totalImagesPullingTime="867.691887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:57:46.823770059 +0000 UTC m=+2877.238508087" watchObservedRunningTime="2026-07-17 18:57:46.82799893 +0000 UTC m=+2877.242736957" Jul 17 18:57:47.520247 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:47.520216 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zv9q4_32593b2d-c97f-4e09-941c-0c2aaa602360/global-pull-secret-syncer/0.log" Jul 17 18:57:47.621687 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:47.621621 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qrr5b_83445c09-b601-4b2a-b503-1d4d7138ee68/konnectivity-agent/0.log" Jul 17 18:57:47.733624 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:47.733598 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-221.ec2.internal_a98b89d541a53d94896a6b1c12baf188/haproxy/0.log" Jul 17 18:57:52.292663 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.292622 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs_938899aa-38b1-42d7-a873-df2b0aef42eb/extract/0.log" Jul 17 18:57:52.323551 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.323427 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs_938899aa-38b1-42d7-a873-df2b0aef42eb/util/0.log" Jul 17 18:57:52.352542 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.352452 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759552bs_938899aa-38b1-42d7-a873-df2b0aef42eb/pull/0.log" Jul 17 18:57:52.385485 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.385461 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m_36c88212-fb53-414b-87a7-a412c0aab5a1/extract/0.log" Jul 17 18:57:52.408202 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.408168 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m_36c88212-fb53-414b-87a7-a412c0aab5a1/util/0.log" Jul 17 18:57:52.434689 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.434655 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q5h2m_36c88212-fb53-414b-87a7-a412c0aab5a1/pull/0.log" Jul 17 18:57:52.477516 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.477477 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n_a6441300-acfd-4e7e-87bc-463e6baccaa3/extract/0.log" Jul 17 18:57:52.500267 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.500191 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n_a6441300-acfd-4e7e-87bc-463e6baccaa3/util/0.log" Jul 17 18:57:52.523780 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.523746 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tbg7n_a6441300-acfd-4e7e-87bc-463e6baccaa3/pull/0.log" Jul 17 18:57:52.552857 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.552778 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz_273b7e4d-641c-4591-a119-cc72059acf9d/extract/0.log" Jul 17 18:57:52.578507 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.578480 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz_273b7e4d-641c-4591-a119-cc72059acf9d/util/0.log" Jul 17 18:57:52.602704 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:52.602670 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qt7sz_273b7e4d-641c-4591-a119-cc72059acf9d/pull/0.log" Jul 17 18:57:53.066997 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:53.066274 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86bd5d5fbc-5tzqt_c22be84f-0691-46e6-a7d1-851775186304/authorino/0.log" Jul 17 18:57:53.105257 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:53.105219 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nt6ns_03f56fa4-6812-4702-aa55-71d3c7814f5e/manager/0.log" Jul 17 18:57:53.134859 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:53.134822 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ftlmm_bdc997a8-42fe-43da-8c59-40ed30a843b7/manager/0.log" Jul 17 18:57:53.217104 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:53.217063 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lc78v_cf5e7b80-d007-4d25-a6ea-f59634b988cd/registry-server/0.log" Jul 17 18:57:53.390483 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:53.390404 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-krrgw_cc1e0fff-6947-4365-878e-282091ed545b/manager/0.log" Jul 17 18:57:53.507986 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:53.505435 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-f7lfn_7e5dd04e-25db-4ef2-9694-2e2f0993ffc3/manager/0.log" Jul 17 18:57:55.582050 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:55.582015 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4wn9w_e319103e-7fd0-4971-ac40-4435bc82eb99/node-exporter/0.log" Jul 17 18:57:55.605106 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:55.605076 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4wn9w_e319103e-7fd0-4971-ac40-4435bc82eb99/kube-rbac-proxy/0.log" Jul 17 18:57:55.631335 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:55.631298 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4wn9w_e319103e-7fd0-4971-ac40-4435bc82eb99/init-textfile/0.log" Jul 17 18:57:57.251899 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.251854 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw"] Jul 17 18:57:57.256713 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.256667 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.273227 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.273114 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw"] Jul 17 18:57:57.292164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.291920 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-podres\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.292164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.292032 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-lib-modules\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.292164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.292056 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748xr\" (UniqueName: \"kubernetes.io/projected/5d388d81-bbfd-4245-8831-508036ad90f9-kube-api-access-748xr\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.292164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.292075 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-proc\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.292164 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.292099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-sys\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394282 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-proc\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-sys\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-podres\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-lib-modules\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748xr\" (UniqueName: \"kubernetes.io/projected/5d388d81-bbfd-4245-8831-508036ad90f9-kube-api-access-748xr\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394683 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-proc\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394747 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-sys\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394861 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-podres\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.396976 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.394974 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d388d81-bbfd-4245-8831-508036ad90f9-lib-modules\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.404777 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.404709 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748xr\" (UniqueName: \"kubernetes.io/projected/5d388d81-bbfd-4245-8831-508036ad90f9-kube-api-access-748xr\") pod \"perf-node-gather-daemonset-w2hhw\" (UID: \"5d388d81-bbfd-4245-8831-508036ad90f9\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.575067 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.574977 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:57.766786 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.766757 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw"] Jul 17 18:57:57.769477 ip-10-0-137-221 kubenswrapper[2570]: W0717 18:57:57.769441 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d388d81_bbfd_4245_8831_508036ad90f9.slice/crio-807468c1a77cfacb58a5d9c95db4c036009275db420dc3906e2928b7d1ac53d6 WatchSource:0}: Error finding container 807468c1a77cfacb58a5d9c95db4c036009275db420dc3906e2928b7d1ac53d6: Status 404 returned error can't find the container with id 807468c1a77cfacb58a5d9c95db4c036009275db420dc3906e2928b7d1ac53d6 Jul 17 18:57:57.913111 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:57.912948 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" event={"ID":"5d388d81-bbfd-4245-8831-508036ad90f9","Type":"ContainerStarted","Data":"807468c1a77cfacb58a5d9c95db4c036009275db420dc3906e2928b7d1ac53d6"} Jul 17 18:57:58.924014 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:58.923288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" event={"ID":"5d388d81-bbfd-4245-8831-508036ad90f9","Type":"ContainerStarted","Data":"3ed3ec6c8892356be6b42e6d5efe45bd66ce60b93eec6a9290eb6917afa67512"} Jul 17 18:57:58.924014 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:58.923671 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:57:58.942419 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:57:58.942352 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" podStartSLOduration=1.9423306390000001 podStartE2EDuration="1.942330639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:57:58.940286865 +0000 UTC m=+2889.355024917" watchObservedRunningTime="2026-07-17 18:57:58.942330639 +0000 UTC m=+2889.357068666" Jul 17 18:58:00.427363 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:00.427326 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f8dn9_d905cff9-94ae-47e1-9c19-490ebe8a40e3/dns/0.log" Jul 17 18:58:00.449532 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:00.449475 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f8dn9_d905cff9-94ae-47e1-9c19-490ebe8a40e3/kube-rbac-proxy/0.log" Jul 17 18:58:00.475454 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:00.475417 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52kz4_5b32749a-dabf-4c93-9dbd-cd822fb5ba85/dns-node-resolver/0.log" Jul 17 18:58:01.133536 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:01.133510 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-btwwl_c0405c6f-6271-4413-aed6-8bf5750e553e/node-ca/0.log" Jul 17 18:58:02.375871 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:02.375833 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-k8lbs_a2bd523b-e875-4aea-b1f4-def4f0875183/kube-auth-proxy/0.log" Jul 17 18:58:02.528805 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:02.528773 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-4kd6z_663d5575-ebca-4529-a9ef-9bbbd6137773/payload-pre-processing/0.log" Jul 17 18:58:03.350774 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:03.350752 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rhq8s_3ac4f76b-38c8-4a90-bb4d-da0959212e44/serve-healthcheck-canary/0.log" Jul 17 18:58:04.022339 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:04.022309 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9ngn_e58f6e97-8740-4378-8965-a3d69b83e938/kube-rbac-proxy/0.log" Jul 17 18:58:04.043937 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:04.043911 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9ngn_e58f6e97-8740-4378-8965-a3d69b83e938/exporter/0.log" Jul 17 18:58:04.066666 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:04.066625 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9ngn_e58f6e97-8740-4378-8965-a3d69b83e938/extractor/0.log" Jul 17 18:58:04.942994 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:04.942817 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-w2hhw" Jul 17 18:58:06.306661 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:06.306627 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tj59h_f57471e8-c7c0-4122-861f-a191ed9c595c/manager/0.log" Jul 17 18:58:06.496365 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:06.496289 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rdvx5_ac647307-f36b-449e-83ea-c4ba30e69624/manager/0.log" Jul 17 18:58:06.619439 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:06.619413 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qbzt6_8ab02e79-2ce3-4217-a8ac-567ae14b3f46/server/0.log" Jul 17 18:58:06.711978 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:06.711732 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hvl2r_22297a5e-a9e3-4741-9466-62f8c3f46878/manager/0.log" Jul 17 18:58:06.802199 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:06.802170 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-2qtd6_54850ee9-6cb1-4e0e-9343-15092fc4581a/manager/0.log" Jul 17 18:58:06.819906 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:06.819881 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-2qtd6_54850ee9-6cb1-4e0e-9343-15092fc4581a/copy-manifests/0.log" Jul 17 18:58:08.323883 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:08.323835 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-48fch_370c3cc5-e768-4c2d-b49c-6e7632e35286/openshift-lws-operator/0.log" Jul 17 18:58:08.332209 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:08.332181 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-k8lbs_a2bd523b-e875-4aea-b1f4-def4f0875183/kube-auth-proxy/0.log" Jul 17 18:58:08.428456 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:08.428418 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-4kd6z_663d5575-ebca-4529-a9ef-9bbbd6137773/payload-pre-processing/0.log" Jul 17 18:58:09.848848 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:09.848822 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-k8lbs_a2bd523b-e875-4aea-b1f4-def4f0875183/kube-auth-proxy/0.log" Jul 17 18:58:09.945728 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:09.945691 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-4kd6z_663d5575-ebca-4529-a9ef-9bbbd6137773/payload-pre-processing/0.log" Jul 17 18:58:14.709247 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:14.709213 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-45tjm_2d4a95c5-11f5-4b4b-a867-6a11cbd3aded/kube-multus/0.log" Jul 17 18:58:14.742447 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:14.742419 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24fhs_e427f223-46d1-4317-a69e-7882713600cd/kube-multus-additional-cni-plugins/0.log" Jul 17 18:58:14.765822 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:14.765790 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24fhs_e427f223-46d1-4317-a69e-7882713600cd/egress-router-binary-copy/0.log" Jul 17 18:58:14.785864 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:14.785840 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24fhs_e427f223-46d1-4317-a69e-7882713600cd/cni-plugins/0.log" Jul 17 18:58:14.809567 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:14.809532 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24fhs_e427f223-46d1-4317-a69e-7882713600cd/bond-cni-plugin/0.log" Jul 17 18:58:14.839256 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:14.839223 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24fhs_e427f223-46d1-4317-a69e-7882713600cd/routeoverride-cni/0.log" Jul 17 18:58:14.862065 ip-10-0-137-221 kubenswrapper[2570]: I0717 18:58:14.862025 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-24fhs_e427f223-46d1-4317-a69e-7882713600cd/whereabouts-cni-bincopy/0.log"