Sep 10 17:09:33.657774 ip-10-0-137-217 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 17:09:33.657786 ip-10-0-137-217 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 17:09:33.657796 ip-10-0-137-217 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 17:09:33.658110 ip-10-0-137-217 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 17:09:43.839392 ip-10-0-137-217 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Sep 10 17:09:43.839407 ip-10-0-137-217 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ce9fe86d721a4d9ea313b2f58a761893 -- Sep 10 17:11:33.893216 ip-10-0-137-217 systemd[1]: Starting Kubernetes Kubelet... Sep 10 17:11:34.304680 ip-10-0-137-217 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 17:11:34.304680 ip-10-0-137-217 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 17:11:34.304680 ip-10-0-137-217 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 17:11:34.304680 ip-10-0-137-217 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 17:11:34.304680 ip-10-0-137-217 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 17:11:34.305707 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.305620 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 17:11:34.309817 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309799 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:11:34.309817 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309818 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309822 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309825 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309829 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309832 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309835 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309838 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309840 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309844 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309848 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309852 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309854 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309859 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309862 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309865 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309868 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309871 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309874 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309893 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309896 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:11:34.309898 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309899 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309901 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309904 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309907 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309910 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309913 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309916 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309918 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309921 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309924 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309926 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309929 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309931 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309934 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309937 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309939 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309942 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309945 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309949 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309951 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:11:34.310390 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309954 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309956 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309958 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309961 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309965 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309968 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309971 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309974 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309977 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309979 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309982 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309984 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309987 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309989 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309993 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309996 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.309998 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310001 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310003 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:11:34.310894 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310006 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310009 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310011 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310014 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310016 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310019 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310021 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310024 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310028 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310031 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310033 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310036 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310038 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310041 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310043 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310046 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310048 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310052 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310054 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310057 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:11:34.311362 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310060 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310062 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310065 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310068 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310070 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310073 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310076 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310080 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310083 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310085 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310088 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310090 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310093 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310095 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310098 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310101 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310103 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310106 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310108 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:11:34.311896 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310792 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310798 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310801 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310804 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310806 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310809 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310812 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310815 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310818 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310821 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310823 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310826 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310829 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310832 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310834 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310837 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310840 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310842 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310845 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310848 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:11:34.312359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310852 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310854 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310857 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310860 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310862 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310865 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310868 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310870 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310873 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310890 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310893 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310896 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310899 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310901 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310904 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310907 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310910 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310912 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310917 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:11:34.312861 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310921 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310924 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310927 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310930 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310933 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310936 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310939 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310941 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310944 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310947 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310950 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310953 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310955 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310958 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310962 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310965 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310968 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310971 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310973 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:11:34.313342 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310976 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310979 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310981 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310984 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310987 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310989 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310992 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310994 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.310998 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311000 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311003 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311006 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311008 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311011 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311014 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311017 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311020 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311022 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311025 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311028 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:11:34.313825 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311031 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311033 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311036 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311038 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311041 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311043 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311046 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311048 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311050 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311053 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311056 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311059 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311061 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311064 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311066 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311068 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311071 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311074 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311076 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311079 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:11:34.314341 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311081 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311450 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311455 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311458 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311461 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311464 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311467 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311470 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311473 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311476 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311479 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311482 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311485 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311487 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311490 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311493 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311496 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311498 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311501 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311503 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:11:34.314849 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311506 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311509 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311511 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311515 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311519 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311522 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311525 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311528 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311531 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311534 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311538 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311544 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311547 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311551 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311554 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311557 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311559 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311562 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:11:34.315359 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311565 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311567 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311571 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311573 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311576 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311579 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311582 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311584 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311587 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311589 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311592 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311594 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311597 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311599 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311602 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311605 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311607 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311609 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311612 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311614 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311617 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:11:34.315927 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311619 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311622 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311624 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311627 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311631 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311633 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311636 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311638 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311641 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311643 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311646 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311648 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311651 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311656 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311658 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311661 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311663 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311666 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311669 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:11:34.316469 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311671 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311674 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311676 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311679 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311681 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311684 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311686 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311689 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311691 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311694 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311697 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311699 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311702 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311704 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311707 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311710 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311712 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311716 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311719 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:11:34.316965 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311721 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311724 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.311726 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311807 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311837 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311844 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311848 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311854 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311857 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311862 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311866 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311869 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311872 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311889 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311893 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311896 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311899 2576 flags.go:64] FLAG: --cgroup-root="" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311902 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311906 2576 flags.go:64] FLAG: --client-ca-file="" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311909 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311912 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311916 2576 flags.go:64] FLAG: --cluster-domain="" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311919 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 17:11:34.317463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311922 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311926 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311929 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311933 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311936 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311939 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311942 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311947 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311950 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311952 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311956 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311958 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311962 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311965 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311968 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311971 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311975 2576 flags.go:64] FLAG: --enable-server="true" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311978 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311983 2576 flags.go:64] FLAG: --event-burst="100" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311986 2576 flags.go:64] FLAG: --event-qps="50" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311989 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311992 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311995 2576 flags.go:64] FLAG: --eviction-hard="" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.311999 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 17:11:34.318038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312002 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312005 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312008 2576 flags.go:64] FLAG: --eviction-soft="" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312010 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312013 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312016 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312019 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312022 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312025 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312027 2576 flags.go:64] FLAG: --feature-gates="" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312031 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312034 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312037 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312040 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312043 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312047 2576 flags.go:64] FLAG: --help="false" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312050 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312053 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312056 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312059 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312062 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312066 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312069 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 17:11:34.318629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312071 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312074 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312077 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312080 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312083 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312086 2576 flags.go:64] FLAG: --kube-reserved="" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312089 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312092 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312095 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312098 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312102 2576 flags.go:64] FLAG: --lock-file="" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312105 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312108 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312110 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312116 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312119 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312122 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312125 2576 flags.go:64] FLAG: --logging-format="text" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312127 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312131 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312134 2576 flags.go:64] FLAG: --manifest-url="" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312137 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312141 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312144 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312150 2576 flags.go:64] FLAG: --max-pods="110" Sep 10 17:11:34.319225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312153 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312156 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312159 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312162 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312165 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312168 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312171 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312178 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312182 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312185 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312188 2576 flags.go:64] FLAG: --pod-cidr="" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312191 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312195 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312198 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312201 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312204 2576 flags.go:64] FLAG: --port="10250" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312207 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312210 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01eed318fd4ba8f88" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312213 2576 flags.go:64] FLAG: --qos-reserved="" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312216 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312219 2576 flags.go:64] FLAG: --register-node="true" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312222 2576 flags.go:64] FLAG: --register-with-taints="" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312226 2576 flags.go:64] FLAG: --registry-burst="10" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312229 2576 flags.go:64] FLAG: --registry-qps="5" Sep 10 17:11:34.319838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312232 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312235 2576 flags.go:64] FLAG: --reserved-memory="" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312238 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312242 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312245 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312248 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312251 2576 flags.go:64] FLAG: --runonce="false" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312255 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312258 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312261 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312264 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312267 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312270 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312273 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312276 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312279 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312282 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312285 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312288 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312291 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312294 2576 flags.go:64] FLAG: --system-cgroups="" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312296 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312302 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312305 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312307 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 17:11:34.320528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312311 2576 flags.go:64] FLAG: --tls-min-version="" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312314 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312317 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312320 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312323 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312326 2576 flags.go:64] FLAG: --v="2" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312330 2576 flags.go:64] FLAG: --version="false" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312334 2576 flags.go:64] FLAG: --vmodule="" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312337 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.312341 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312462 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312465 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312468 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312471 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312475 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312477 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312481 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312484 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312486 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312489 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312492 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312494 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:11:34.321173 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312497 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312499 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312502 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312505 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312507 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312510 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312512 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312515 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312518 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312520 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312523 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312525 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312528 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312531 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312533 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312536 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312539 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312541 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312544 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312546 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:11:34.321731 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312549 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312552 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312554 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312557 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312560 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312562 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312566 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312569 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312573 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312576 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312579 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312581 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312584 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312587 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312589 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312592 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312595 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312597 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312600 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:11:34.322247 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312602 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312605 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312607 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312610 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312612 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312615 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312618 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312621 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312623 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312626 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312629 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312632 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312634 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312637 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312639 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312642 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312645 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312648 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312650 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312654 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:11:34.322707 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312656 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312659 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312662 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312664 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312667 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312671 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312675 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312678 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312680 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312684 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312686 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312689 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312692 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312694 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312697 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312700 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312704 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312707 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312709 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:11:34.323314 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312713 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312715 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312718 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312721 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312723 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312726 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312728 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312731 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.312734 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.313291 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.320421 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.320438 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.320452 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.320460 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320520 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:11:34.323784 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320525 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320528 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320531 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320534 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320537 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320540 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320543 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320546 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320548 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320551 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320554 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320557 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320559 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320562 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320565 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320567 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320570 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320572 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320575 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320578 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:11:34.324174 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320580 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320582 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320585 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320587 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320590 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320592 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320595 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320599 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320611 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320614 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320617 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320619 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320623 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320627 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320630 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320632 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320635 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320637 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320640 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:11:34.324708 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320642 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320645 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320647 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320650 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320652 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320655 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320657 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320660 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320662 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320665 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320667 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320670 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320672 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320675 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320677 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320680 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320682 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320684 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320688 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:11:34.325187 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320691 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320694 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320697 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320706 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320709 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320711 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320714 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320717 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320719 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320722 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320724 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320727 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320729 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320732 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320734 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320737 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320740 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320742 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320745 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320748 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:11:34.325661 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320750 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320753 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320755 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320757 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320760 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320763 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320765 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320768 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320770 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320772 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320775 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320778 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320781 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320783 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320786 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320788 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320796 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320799 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320801 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:11:34.326384 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320804 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.320809 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320984 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320989 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320993 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320996 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.320999 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321001 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321004 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321006 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321009 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321012 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321015 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321017 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321020 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:11:34.327148 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321023 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321025 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321028 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321030 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321033 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321035 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321038 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321040 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321043 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321046 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321048 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321051 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321053 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321055 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321058 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321066 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321069 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321071 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321074 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321076 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:11:34.327662 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321079 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321082 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321084 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321087 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321090 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321092 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321095 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321097 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321099 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321102 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321105 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321107 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321110 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321112 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321115 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321117 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321120 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321123 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321125 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321128 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:11:34.328242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321130 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321133 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321135 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321138 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321140 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321143 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321145 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321148 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321155 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321158 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321160 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321163 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321165 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321168 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321170 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321173 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321176 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321178 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321181 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321184 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:11:34.328725 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321186 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321189 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321192 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321194 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321197 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321200 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321203 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321207 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321210 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321213 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321216 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321219 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321221 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321224 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321227 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321230 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321232 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321235 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:11:34.329236 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321239 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321243 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321246 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321255 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321258 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321261 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321264 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:34.321266 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.321271 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.321395 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.324113 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.325122 2576 server.go:1019] "Starting client certificate rotation" Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.325239 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 17:11:34.329679 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.325298 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 17:11:34.330027 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.329546 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 17:11:34.331445 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.331429 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 17:11:34.342765 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.342743 2576 log.go:25] "Validated CRI v1 runtime API" Sep 10 17:11:34.346963 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.346949 2576 log.go:25] "Validated CRI v1 image API" Sep 10 17:11:34.347845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.347831 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 17:11:34.351293 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.351274 2576 fs.go:135] Filesystem UUIDs: map[1a9c052c-a93e-42e3-bc60-fdb01da84411:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bff6e647-4f1e-4c80-a91f-a89276e7f008:/dev/nvme0n1p3] Sep 10 17:11:34.351350 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.351292 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 17:11:34.357617 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.357498 2576 manager.go:216] Machine: {Timestamp:2026-09-10 17:11:34.35537927 +0000 UTC m=+0.336459259 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100207 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e47aa1607b0a598c7f0a72012552a SystemUUID:ec2e47aa-1607-b0a5-98c7-f0a72012552a BootID:ce9fe86d-721a-4d9e-a313-b2f58a761893 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cb:fa:b8:3b:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cb:fa:b8:3b:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:30:d6:c6:3a:25 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 17:11:34.357617 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.357613 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 17:11:34.357731 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.357702 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 17:11:34.359197 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.359179 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 17:11:34.360706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.360679 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 17:11:34.360835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.360710 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-217.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 17:11:34.362326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.362315 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 17:11:34.362326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.362327 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 17:11:34.362392 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.362335 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 17:11:34.363095 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.363084 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 17:11:34.364265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.364255 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 17:11:34.367835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.367824 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 17:11:34.367982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.367972 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 17:11:34.368014 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.368003 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 10 17:11:34.368014 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.368013 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 17:11:34.368074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.368027 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 17:11:34.368074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.368048 2576 kubelet.go:398] "Adding apiserver pod source" Sep 10 17:11:34.368074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.368057 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 17:11:34.369594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.369575 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 17:11:34.369662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.369607 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 17:11:34.372567 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.372547 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 17:11:34.374831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.374818 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 17:11:34.374900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.374847 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 17:11:34.375016 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375004 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 17:11:34.375049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375028 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 17:11:34.375049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375040 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 17:11:34.375102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375050 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 17:11:34.375102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375061 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 17:11:34.375102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375070 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 17:11:34.375658 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375647 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 17:11:34.375658 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375658 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 17:11:34.375726 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375665 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 17:11:34.375726 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375670 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 17:11:34.375726 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375679 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 17:11:34.375726 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.375687 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 17:11:34.376519 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.376500 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 17:11:34.376519 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.376513 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 17:11:34.380377 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.380361 2576 server.go:1268] "Started kubelet" Sep 10 17:11:34.380476 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.380441 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 17:11:34.380532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.380442 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 17:11:34.380532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.380525 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 17:11:34.381398 ip-10-0-137-217 systemd[1]: Started Kubernetes Kubelet. Sep 10 17:11:34.382104 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.382030 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 17:11:34.382185 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.382139 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 10 17:11:34.382463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.382433 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-217.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 17:11:34.382549 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.382502 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 17:11:34.382636 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.382543 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 17:11:34.386760 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.386673 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 17:11:34.388074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.388054 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 17:11:34.388499 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.387443 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-217.ec2.internal.18d404dbe04bb2bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-217.ec2.internal,UID:ip-10-0-137-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-217.ec2.internal,},FirstTimestamp:2026-09-10 17:11:34.380335807 +0000 UTC m=+0.361415797,LastTimestamp:2026-09-10 17:11:34.380335807 +0000 UTC m=+0.361415797,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-217.ec2.internal,}" Sep 10 17:11:34.388592 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.388506 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 17:11:34.388643 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.388596 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:34.389172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389154 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 17:11:34.389172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389172 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 17:11:34.389475 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389459 2576 factory.go:55] Registering systemd factory Sep 10 17:11:34.389536 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389479 2576 factory.go:223] Registration of the systemd container factory successfully Sep 10 17:11:34.389716 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389700 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 17:11:34.389785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389728 2576 factory.go:153] Registering CRI-O factory Sep 10 17:11:34.389785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389744 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 10 17:11:34.389785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389756 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 10 17:11:34.389785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389745 2576 factory.go:223] Registration of the crio container factory successfully Sep 10 17:11:34.389943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389800 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 17:11:34.389943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389817 2576 factory.go:103] Registering Raw factory Sep 10 17:11:34.389943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.389827 2576 manager.go:1196] Started watching for new ooms in manager Sep 10 17:11:34.390215 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.390201 2576 manager.go:349] Starting recovery of all containers Sep 10 17:11:34.394049 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.394012 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 17:11:34.394148 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.394132 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 17:11:34.402480 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.402329 2576 manager.go:354] Recovery completed Sep 10 17:11:34.407888 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.407863 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:11:34.409974 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.409951 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:11:34.410038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.409997 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:11:34.410038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.410011 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:11:34.410622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.410607 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 17:11:34.410622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.410619 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 17:11:34.410721 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.410634 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 17:11:34.410721 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.410649 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:11:34.411371 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.411352 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:11:34.411456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.411386 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:11:34.411456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.411400 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:11:34.412174 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.412106 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-217.ec2.internal.18d404dbe20fecfe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-217.ec2.internal,UID:ip-10-0-137-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-217.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-217.ec2.internal,},FirstTimestamp:2026-09-10 17:11:34.40997299 +0000 UTC m=+0.391052986,LastTimestamp:2026-09-10 17:11:34.40997299 +0000 UTC m=+0.391052986,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-217.ec2.internal,}" Sep 10 17:11:34.412280 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.412224 2576 policy_none.go:49] "None policy: Start" Sep 10 17:11:34.412280 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.412239 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 17:11:34.412280 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.412248 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 17:11:34.412908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.412871 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9mw5l" Sep 10 17:11:34.413267 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.413255 2576 policy_none.go:47] "Start" Sep 10 17:11:34.421041 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.421022 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9mw5l" Sep 10 17:11:34.421186 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.421083 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-217.ec2.internal.18d404dbe21064d5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-217.ec2.internal,UID:ip-10-0-137-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-217.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-217.ec2.internal,},FirstTimestamp:2026-09-10 17:11:34.410003669 +0000 UTC m=+0.391083662,LastTimestamp:2026-09-10 17:11:34.410003669 +0000 UTC m=+0.391083662,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-217.ec2.internal,}" Sep 10 17:11:34.438539 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.438524 2576 manager.go:337] "Starting Device Plugin manager" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.438564 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.438577 2576 server.go:87] "Starting device plugin registration server" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.438910 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.438926 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.439103 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.439172 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.439181 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.439941 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 17:11:34.453956 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.439995 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:34.478621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.478588 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 17:11:34.479826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.479809 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 17:11:34.479895 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.479866 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 17:11:34.479944 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.479908 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 17:11:34.480014 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.479971 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 17:11:34.482369 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.482342 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:11:34.539608 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.539582 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:11:34.540892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.540863 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:11:34.540997 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.540910 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:11:34.540997 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.540926 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:11:34.540997 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.540956 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.551797 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.551775 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.551797 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.551798 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-217.ec2.internal\": node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:34.580960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.580867 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal"] Sep 10 17:11:34.581050 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.580958 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:11:34.582672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.582655 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:11:34.582769 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.582683 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:11:34.582769 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.582693 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:11:34.584044 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584031 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:11:34.584184 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.584230 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584197 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:11:34.584772 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584753 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:11:34.584772 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584764 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:11:34.584910 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584781 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:11:34.584910 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584785 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:11:34.584910 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584791 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:11:34.584910 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.584800 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:11:34.585116 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.585060 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:34.586011 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.585994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.586105 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.586022 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:11:34.587313 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.587299 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:11:34.587410 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.587326 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:11:34.587410 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.587339 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:11:34.590540 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.590523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f318e2a8b58477e08e99d42f801553c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal\" (UID: \"f318e2a8b58477e08e99d42f801553c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.590649 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.590552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f318e2a8b58477e08e99d42f801553c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal\" (UID: \"f318e2a8b58477e08e99d42f801553c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.590649 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.590576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56c1d6772edd98b5770557256f012f4e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-217.ec2.internal\" (UID: \"56c1d6772edd98b5770557256f012f4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.618663 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.618639 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-217.ec2.internal\" not found" node="ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.623144 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.623128 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-217.ec2.internal\" not found" node="ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.685762 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.685731 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:34.691078 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.691055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56c1d6772edd98b5770557256f012f4e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-217.ec2.internal\" (UID: \"56c1d6772edd98b5770557256f012f4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.691162 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.691085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f318e2a8b58477e08e99d42f801553c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal\" (UID: \"f318e2a8b58477e08e99d42f801553c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.691162 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.691102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f318e2a8b58477e08e99d42f801553c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal\" (UID: \"f318e2a8b58477e08e99d42f801553c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.691231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.691149 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56c1d6772edd98b5770557256f012f4e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-217.ec2.internal\" (UID: \"56c1d6772edd98b5770557256f012f4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.691231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.691172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f318e2a8b58477e08e99d42f801553c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal\" (UID: \"f318e2a8b58477e08e99d42f801553c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.691231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.691197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f318e2a8b58477e08e99d42f801553c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal\" (UID: \"f318e2a8b58477e08e99d42f801553c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.785890 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.785846 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:34.886723 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.886651 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:34.921118 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.921083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.925961 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:34.925944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" Sep 10 17:11:34.987676 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:34.987645 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:35.088164 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:35.088118 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:35.194263 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:35.194171 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:35.294433 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:35.294393 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-217.ec2.internal\" not found" Sep 10 17:11:35.325044 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.325022 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 17:11:35.325461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.325165 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 17:11:35.393068 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.393034 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:11:35.423892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.423835 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 17:06:35 +0000 UTC" deadline="2028-02-09 18:22:30.454010929 +0000 UTC" Sep 10 17:11:35.423892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.423871 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12409h10m55.030143796s" Sep 10 17:11:35.449939 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.449859 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:11:35.489560 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.489536 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" Sep 10 17:11:35.498443 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.498421 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 17:11:35.499387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.499374 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" Sep 10 17:11:35.511938 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.511916 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 17:11:35.744102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.744032 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:11:35.758180 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:35.758134 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c1d6772edd98b5770557256f012f4e.slice/crio-ba6b1e2ed910ce6dfa244cffbc5732b4c6b48483c52c85ed2842d75922b6477f WatchSource:0}: Error finding container ba6b1e2ed910ce6dfa244cffbc5732b4c6b48483c52c85ed2842d75922b6477f: Status 404 returned error can't find the container with id ba6b1e2ed910ce6dfa244cffbc5732b4c6b48483c52c85ed2842d75922b6477f Sep 10 17:11:35.758414 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:35.758391 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf318e2a8b58477e08e99d42f801553c7.slice/crio-783ea3e4c17644c393b5ac612ab34ca313f67e8c6ae1710de8204351205e2955 WatchSource:0}: Error finding container 783ea3e4c17644c393b5ac612ab34ca313f67e8c6ae1710de8204351205e2955: Status 404 returned error can't find the container with id 783ea3e4c17644c393b5ac612ab34ca313f67e8c6ae1710de8204351205e2955 Sep 10 17:11:35.764160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:35.764140 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:11:36.372248 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.372215 2576 apiserver.go:52] "Watching apiserver" Sep 10 17:11:36.372716 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.372290 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:11:36.381539 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.381510 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 17:11:36.382040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.382008 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-xqf2n","openshift-ovn-kubernetes/ovnkube-node-tvskl","kube-system/global-pull-secret-syncer-8r6bk","kube-system/konnectivity-agent-b4rjd","kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb","openshift-image-registry/node-ca-d2rkj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal","openshift-multus/multus-additional-cni-plugins-9r256","openshift-multus/multus-lgfj2","openshift-cluster-node-tuning-operator/tuned-9khk5","openshift-dns/node-resolver-d9247","openshift-multus/network-metrics-daemon-555sb","openshift-network-diagnostics/network-check-target-t5lgb"] Sep 10 17:11:36.383909 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.383876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.385507 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.385250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.386434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.386408 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4g7cn\"" Sep 10 17:11:36.386595 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.386576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.386679 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.386654 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:36.386975 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.386943 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 17:11:36.387076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.386980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 17:11:36.387076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.387009 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 17:11:36.387627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.387606 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:11:36.387786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.387762 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5t7wm\"" Sep 10 17:11:36.387856 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.387816 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 17:11:36.388443 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.388234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.389094 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.389073 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 17:11:36.389581 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.389557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:36.389694 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.389638 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:36.391080 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.391060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.391430 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.391408 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 17:11:36.392334 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.392306 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 17:11:36.392434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.392336 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jkgw6\"" Sep 10 17:11:36.393506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.393481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 17:11:36.393673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.393648 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 17:11:36.394145 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.394016 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rncgg\"" Sep 10 17:11:36.396380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.394354 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 17:11:36.396380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.394368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.397071 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.397046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 17:11:36.397320 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.397278 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 17:11:36.397389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.397353 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x4tfw\"" Sep 10 17:11:36.397577 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.397556 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 17:11:36.397688 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.397661 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 17:11:36.397782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.397757 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 17:11:36.398059 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.398034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:36.398153 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.398106 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:36.399714 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.399694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.400205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.400182 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 17:11:36.401212 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.401192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.402199 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6feefb4-04cc-497b-870f-609aab9359f9-serviceca\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.402199 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-kubernetes\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402322 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-systemd\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402322 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-run\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402322 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-device-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.402322 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-host\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402341 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af136ee8-5433-4faa-868a-83038e568060-agent-certs\") pod \"konnectivity-agent-b4rjd\" (UID: \"af136ee8-5433-4faa-868a-83038e568060\") " pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-modprobe-d\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-socket-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402472 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwsf\" (UniqueName: \"kubernetes.io/projected/485fe935-b6ec-4702-87a5-a406108d9032-kube-api-access-sdwsf\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.402531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30c0389e-f4f6-45a9-b319-06639ac1a224-tmp\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-kubelet-config\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-sys-fs\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6feefb4-04cc-497b-870f-609aab9359f9-host\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402645 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-sys\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402685 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6m6lz\"" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af136ee8-5433-4faa-868a-83038e568060-konnectivity-ca\") pod \"konnectivity-agent-b4rjd\" (UID: \"af136ee8-5433-4faa-868a-83038e568060\") " pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwphz\" (UniqueName: \"kubernetes.io/projected/f6feefb4-04cc-497b-870f-609aab9359f9-kube-api-access-nwphz\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysconfig\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-lib-modules\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402834 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 17:11:36.402858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-var-lib-kubelet\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402877 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-tuned\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-dbus\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.402972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysctl-d\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysctl-conf\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgb9\" (UniqueName: \"kubernetes.io/projected/30c0389e-f4f6-45a9-b319-06639ac1a224-kube-api-access-tvgb9\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.403470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-registration-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.403907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 17:11:36.403907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403640 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 17:11:36.403907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.403879 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g8q78\"" Sep 10 17:11:36.404095 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.404070 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.405905 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.405558 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 17:11:36.405905 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.405724 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 17:11:36.406285 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.406265 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhbt\"" Sep 10 17:11:36.406571 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.406551 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 17:11:36.406724 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.406595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:11:36.407092 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.407071 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 17:11:36.407185 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.407140 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7p7jd\"" Sep 10 17:11:36.429978 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.429948 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dr9c4" Sep 10 17:11:36.437457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.437434 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dr9c4" Sep 10 17:11:36.486574 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.486521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" event={"ID":"f318e2a8b58477e08e99d42f801553c7","Type":"ContainerStarted","Data":"783ea3e4c17644c393b5ac612ab34ca313f67e8c6ae1710de8204351205e2955"} Sep 10 17:11:36.492379 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.490458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" event={"ID":"56c1d6772edd98b5770557256f012f4e","Type":"ContainerStarted","Data":"ba6b1e2ed910ce6dfa244cffbc5732b4c6b48483c52c85ed2842d75922b6477f"} Sep 10 17:11:36.492379 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.491650 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6feefb4-04cc-497b-870f-609aab9359f9-serviceca\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-systemd\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-run\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fbd2036-b001-419a-a567-ac76cd7a7802-iptables-alerter-script\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-ovn\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-cnibin\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-cni-bin\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-host\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503653 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-systemd\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af136ee8-5433-4faa-868a-83038e568060-agent-certs\") pod \"konnectivity-agent-b4rjd\" (UID: \"af136ee8-5433-4faa-868a-83038e568060\") " pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-run\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbz6h\" (UniqueName: \"kubernetes.io/projected/6fbd2036-b001-419a-a567-ac76cd7a7802-kube-api-access-nbz6h\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-cni-bin\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-ovnkube-config\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-cni-multus\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.504845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503755 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-host\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjh5l\" (UniqueName: \"kubernetes.io/projected/4721421f-ab96-45e5-8503-ccc0a9a3f83f-kube-api-access-qjh5l\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-systemd-units\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.503974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-etc-selinux\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6feefb4-04cc-497b-870f-609aab9359f9-host\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6feefb4-04cc-497b-870f-609aab9359f9-host\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af136ee8-5433-4faa-868a-83038e568060-konnectivity-ca\") pod \"konnectivity-agent-b4rjd\" (UID: \"af136ee8-5433-4faa-868a-83038e568060\") " pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40439c75-55c7-42f5-9d84-6b380daf2e05-hosts-file\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwphz\" (UniqueName: \"kubernetes.io/projected/f6feefb4-04cc-497b-870f-609aab9359f9-kube-api-access-nwphz\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysconfig\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-var-lib-kubelet\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-dbus\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.505900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysconfig\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-kubelet\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-kubelet\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504346 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-dbus\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-var-lib-kubelet\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysctl-d\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgb9\" (UniqueName: \"kubernetes.io/projected/30c0389e-f4f6-45a9-b319-06639ac1a224-kube-api-access-tvgb9\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-registration-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-system-cni-dir\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysctl-d\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-etc-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-env-overrides\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-netns\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-multus-certs\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-kubernetes\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.506709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-slash\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504616 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-var-lib-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lm76\" (UniqueName: \"kubernetes.io/projected/ea205362-1082-4392-8909-ee82fc215754-kube-api-access-8lm76\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af136ee8-5433-4faa-868a-83038e568060-konnectivity-ca\") pod \"konnectivity-agent-b4rjd\" (UID: \"af136ee8-5433-4faa-868a-83038e568060\") " pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-device-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fbd2036-b001-419a-a567-ac76cd7a7802-host-slash\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-device-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gpn\" (UniqueName: \"kubernetes.io/projected/a2c95e10-40c9-4da0-be08-744fb5e1cab8-kube-api-access-22gpn\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-registration-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67b82be8-09d6-4631-bff0-14481c445847-multus-daemon-config\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-kubernetes\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-etc-kubernetes\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j7d5\" (UniqueName: \"kubernetes.io/projected/67b82be8-09d6-4631-bff0-14481c445847-kube-api-access-5j7d5\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.504986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-modprobe-d\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-node-log\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.507503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-modprobe-d\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-run-ovn-kubernetes\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6feefb4-04cc-497b-870f-609aab9359f9-serviceca\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea205362-1082-4392-8909-ee82fc215754-ovn-node-metrics-cert\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-ovnkube-script-lib\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-k8s-cni-cncf-io\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-socket-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwsf\" (UniqueName: \"kubernetes.io/projected/485fe935-b6ec-4702-87a5-a406108d9032-kube-api-access-sdwsf\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30c0389e-f4f6-45a9-b319-06639ac1a224-tmp\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-kubelet-config\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-socket-dir\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cnibin\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-kubelet-config\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-run-netns\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.509620 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.505544 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.505629 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret podName:0f7c913b-fedc-4ca8-9d95-fa73663da5d4 nodeName:}" failed. No retries permitted until 2026-09-10 17:11:37.005606052 +0000 UTC m=+2.986686046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret") pod "global-pull-secret-syncer-8r6bk" (UID: "0f7c913b-fedc-4ca8-9d95-fa73663da5d4") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-sys-fs\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-sys\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/485fe935-b6ec-4702-87a5-a406108d9032-sys-fs\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40439c75-55c7-42f5-9d84-6b380daf2e05-tmp-dir\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-log-socket\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-cni-netd\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-cni-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67b82be8-09d6-4631-bff0-14481c445847-cni-binary-copy\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-socket-dir-parent\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-lib-modules\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-tuned\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-sys\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72z6c\" (UniqueName: \"kubernetes.io/projected/40439c75-55c7-42f5-9d84-6b380daf2e05-kube-api-access-72z6c\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-hostroot\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.510530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.505996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysctl-conf\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.506022 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.506047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-systemd\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.506052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-lib-modules\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.506071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-system-cni-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.506103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-conf-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.506285 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-sysctl-conf\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.509680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30c0389e-f4f6-45a9-b319-06639ac1a224-tmp\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.509829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30c0389e-f4f6-45a9-b319-06639ac1a224-etc-tuned\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.510037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af136ee8-5433-4faa-868a-83038e568060-agent-certs\") pod \"konnectivity-agent-b4rjd\" (UID: \"af136ee8-5433-4faa-868a-83038e568060\") " pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.510530 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.510549 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.510559 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pmc4z for pod openshift-network-diagnostics/network-check-target-t5lgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:36.511372 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.510623 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z podName:66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d nodeName:}" failed. No retries permitted until 2026-09-10 17:11:37.010608046 +0000 UTC m=+2.991688057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pmc4z" (UniqueName: "kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z") pod "network-check-target-t5lgb" (UID: "66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:36.514216 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.514194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgb9\" (UniqueName: \"kubernetes.io/projected/30c0389e-f4f6-45a9-b319-06639ac1a224-kube-api-access-tvgb9\") pod \"tuned-9khk5\" (UID: \"30c0389e-f4f6-45a9-b319-06639ac1a224\") " pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.514796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.514753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwphz\" (UniqueName: \"kubernetes.io/projected/f6feefb4-04cc-497b-870f-609aab9359f9-kube-api-access-nwphz\") pod \"node-ca-d2rkj\" (UID: \"f6feefb4-04cc-497b-870f-609aab9359f9\") " pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.516078 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.515918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwsf\" (UniqueName: \"kubernetes.io/projected/485fe935-b6ec-4702-87a5-a406108d9032-kube-api-access-sdwsf\") pod \"aws-ebs-csi-driver-node-9q9fb\" (UID: \"485fe935-b6ec-4702-87a5-a406108d9032\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.606687 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-socket-dir-parent\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.606687 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-socket-dir-parent\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72z6c\" (UniqueName: \"kubernetes.io/projected/40439c75-55c7-42f5-9d84-6b380daf2e05-kube-api-access-72z6c\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-hostroot\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-systemd\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-system-cni-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-hostroot\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-conf-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-conf-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.606930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fbd2036-b001-419a-a567-ac76cd7a7802-iptables-alerter-script\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.606978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-ovn\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-cnibin\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-cni-bin\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbz6h\" (UniqueName: \"kubernetes.io/projected/6fbd2036-b001-419a-a567-ac76cd7a7802-kube-api-access-nbz6h\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-cni-bin\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607104 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-ovnkube-config\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-cni-multus\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-systemd\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjh5l\" (UniqueName: \"kubernetes.io/projected/4721421f-ab96-45e5-8503-ccc0a9a3f83f-kube-api-access-qjh5l\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-cni-bin\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-systemd-units\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-cni-bin\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:36.607405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-cni-multus\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40439c75-55c7-42f5-9d84-6b380daf2e05-hosts-file\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-kubelet\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-ovn\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-cnibin\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-kubelet\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-system-cni-dir\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-etc-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-env-overrides\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-netns\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fbd2036-b001-419a-a567-ac76cd7a7802-iptables-alerter-script\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-multus-certs\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-slash\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-slash\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607747 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-system-cni-dir\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-etc-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607808 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-ovnkube-config\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.608254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-var-lib-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lm76\" (UniqueName: \"kubernetes.io/projected/ea205362-1082-4392-8909-ee82fc215754-kube-api-access-8lm76\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-netns\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fbd2036-b001-419a-a567-ac76cd7a7802-host-slash\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-systemd-units\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40439c75-55c7-42f5-9d84-6b380daf2e05-hosts-file\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-kubelet\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-system-cni-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.607955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-var-lib-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-run-openvswitch\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-env-overrides\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-var-lib-kubelet\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.608189 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:36.608248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:11:37.108229338 +0000 UTC m=+3.089309322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:36.609231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-multus-certs\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gpn\" (UniqueName: \"kubernetes.io/projected/a2c95e10-40c9-4da0-be08-744fb5e1cab8-kube-api-access-22gpn\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67b82be8-09d6-4631-bff0-14481c445847-multus-daemon-config\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-etc-kubernetes\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j7d5\" (UniqueName: \"kubernetes.io/projected/67b82be8-09d6-4631-bff0-14481c445847-kube-api-access-5j7d5\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-node-log\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-run-ovn-kubernetes\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea205362-1082-4392-8909-ee82fc215754-ovn-node-metrics-cert\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608608 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fbd2036-b001-419a-a567-ac76cd7a7802-host-slash\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-ovnkube-script-lib\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-k8s-cni-cncf-io\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cnibin\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-host-run-k8s-cni-cncf-io\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-run-netns\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40439c75-55c7-42f5-9d84-6b380daf2e05-tmp-dir\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.609999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-log-socket\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-cni-netd\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-node-log\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-cni-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67b82be8-09d6-4631-bff0-14481c445847-cni-binary-copy\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.608953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-run-ovn-kubernetes\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-run-netns\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609128 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea205362-1082-4392-8909-ee82fc215754-ovnkube-script-lib\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-etc-kubernetes\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-log-socket\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40439c75-55c7-42f5-9d84-6b380daf2e05-tmp-dir\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c95e10-40c9-4da0-be08-744fb5e1cab8-cnibin\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea205362-1082-4392-8909-ee82fc215754-host-cni-netd\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67b82be8-09d6-4631-bff0-14481c445847-multus-cni-dir\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67b82be8-09d6-4631-bff0-14481c445847-multus-daemon-config\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.609553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67b82be8-09d6-4631-bff0-14481c445847-cni-binary-copy\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.610790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.610074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c95e10-40c9-4da0-be08-744fb5e1cab8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.612017 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.611984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea205362-1082-4392-8909-ee82fc215754-ovn-node-metrics-cert\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.617114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.617084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjh5l\" (UniqueName: \"kubernetes.io/projected/4721421f-ab96-45e5-8503-ccc0a9a3f83f-kube-api-access-qjh5l\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:36.618237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.618189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbz6h\" (UniqueName: \"kubernetes.io/projected/6fbd2036-b001-419a-a567-ac76cd7a7802-kube-api-access-nbz6h\") pod \"iptables-alerter-xqf2n\" (UID: \"6fbd2036-b001-419a-a567-ac76cd7a7802\") " pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.620419 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.620371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lm76\" (UniqueName: \"kubernetes.io/projected/ea205362-1082-4392-8909-ee82fc215754-kube-api-access-8lm76\") pod \"ovnkube-node-tvskl\" (UID: \"ea205362-1082-4392-8909-ee82fc215754\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.622158 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.622120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gpn\" (UniqueName: \"kubernetes.io/projected/a2c95e10-40c9-4da0-be08-744fb5e1cab8-kube-api-access-22gpn\") pod \"multus-additional-cni-plugins-9r256\" (UID: \"a2c95e10-40c9-4da0-be08-744fb5e1cab8\") " pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.623168 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.623091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j7d5\" (UniqueName: \"kubernetes.io/projected/67b82be8-09d6-4631-bff0-14481c445847-kube-api-access-5j7d5\") pod \"multus-lgfj2\" (UID: \"67b82be8-09d6-4631-bff0-14481c445847\") " pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.624624 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.624570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72z6c\" (UniqueName: \"kubernetes.io/projected/40439c75-55c7-42f5-9d84-6b380daf2e05-kube-api-access-72z6c\") pod \"node-resolver-d9247\" (UID: \"40439c75-55c7-42f5-9d84-6b380daf2e05\") " pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.698811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.698763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d2rkj" Sep 10 17:11:36.706997 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.706737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9khk5" Sep 10 17:11:36.712134 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.712088 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6feefb4_04cc_497b_870f_609aab9359f9.slice/crio-a68df7e4b56fadb8fa518af033342d9db3a083d950a19ab612d0c094cb3b487e WatchSource:0}: Error finding container a68df7e4b56fadb8fa518af033342d9db3a083d950a19ab612d0c094cb3b487e: Status 404 returned error can't find the container with id a68df7e4b56fadb8fa518af033342d9db3a083d950a19ab612d0c094cb3b487e Sep 10 17:11:36.718090 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.718055 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c0389e_f4f6_45a9_b319_06639ac1a224.slice/crio-de5780611edea2689553be1b670caf3934907b99117c588da780cf39f99607ed WatchSource:0}: Error finding container de5780611edea2689553be1b670caf3934907b99117c588da780cf39f99607ed: Status 404 returned error can't find the container with id de5780611edea2689553be1b670caf3934907b99117c588da780cf39f99607ed Sep 10 17:11:36.718775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.718752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:36.725347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.725325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" Sep 10 17:11:36.728351 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.728327 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf136ee8_5433_4faa_868a_83038e568060.slice/crio-f5737562b939d204837800101c9c44bb1b051773c4b779efd784fb86911d4247 WatchSource:0}: Error finding container f5737562b939d204837800101c9c44bb1b051773c4b779efd784fb86911d4247: Status 404 returned error can't find the container with id f5737562b939d204837800101c9c44bb1b051773c4b779efd784fb86911d4247 Sep 10 17:11:36.733174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.733015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r256" Sep 10 17:11:36.737091 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.737065 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485fe935_b6ec_4702_87a5_a406108d9032.slice/crio-2c1b2a2a3759fba34759e766a513302ef28c94f319c03ebef8213c8684d6ab12 WatchSource:0}: Error finding container 2c1b2a2a3759fba34759e766a513302ef28c94f319c03ebef8213c8684d6ab12: Status 404 returned error can't find the container with id 2c1b2a2a3759fba34759e766a513302ef28c94f319c03ebef8213c8684d6ab12 Sep 10 17:11:36.742170 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.742147 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:11:36.745993 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.745961 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c95e10_40c9_4da0_be08_744fb5e1cab8.slice/crio-5da507621ce8929350c36c0aaa2d1f916f5b0c671f967bb8801d3da8e184b60a WatchSource:0}: Error finding container 5da507621ce8929350c36c0aaa2d1f916f5b0c671f967bb8801d3da8e184b60a: Status 404 returned error can't find the container with id 5da507621ce8929350c36c0aaa2d1f916f5b0c671f967bb8801d3da8e184b60a Sep 10 17:11:36.750107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.749847 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lgfj2" Sep 10 17:11:36.751475 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.751451 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea205362_1082_4392_8909_ee82fc215754.slice/crio-f3f36fe73dab0c1352ed38e6fe4d70712de15bb997c936544e2acb882f141b49 WatchSource:0}: Error finding container f3f36fe73dab0c1352ed38e6fe4d70712de15bb997c936544e2acb882f141b49: Status 404 returned error can't find the container with id f3f36fe73dab0c1352ed38e6fe4d70712de15bb997c936544e2acb882f141b49 Sep 10 17:11:36.759207 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.759038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqf2n" Sep 10 17:11:36.759363 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.759330 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b82be8_09d6_4631_bff0_14481c445847.slice/crio-a12d215dfea36ebbdd260a9c8a5db1480e3a32a4080f221d2822b8587e326656 WatchSource:0}: Error finding container a12d215dfea36ebbdd260a9c8a5db1480e3a32a4080f221d2822b8587e326656: Status 404 returned error can't find the container with id a12d215dfea36ebbdd260a9c8a5db1480e3a32a4080f221d2822b8587e326656 Sep 10 17:11:36.766775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:36.766752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d9247" Sep 10 17:11:36.770463 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.770315 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbd2036_b001_419a_a567_ac76cd7a7802.slice/crio-da7ba14e69c6fb1628f7413e506de3fd063c2c1ffb0cbbacafc4f3ed68573abf WatchSource:0}: Error finding container da7ba14e69c6fb1628f7413e506de3fd063c2c1ffb0cbbacafc4f3ed68573abf: Status 404 returned error can't find the container with id da7ba14e69c6fb1628f7413e506de3fd063c2c1ffb0cbbacafc4f3ed68573abf Sep 10 17:11:36.777917 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:11:36.777861 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40439c75_55c7_42f5_9d84_6b380daf2e05.slice/crio-e1623e1d7a6e5d099bbfc393c532fa087ebc4bb4d4aa1ef047ba1479743e00cd WatchSource:0}: Error finding container e1623e1d7a6e5d099bbfc393c532fa087ebc4bb4d4aa1ef047ba1479743e00cd: Status 404 returned error can't find the container with id e1623e1d7a6e5d099bbfc393c532fa087ebc4bb4d4aa1ef047ba1479743e00cd Sep 10 17:11:37.012507 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.012413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:37.012685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.012507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:37.012685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.012623 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:11:37.012685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.012650 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:11:37.012685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.012664 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pmc4z for pod openshift-network-diagnostics/network-check-target-t5lgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:37.012947 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.012620 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:37.012947 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.012722 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z podName:66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d nodeName:}" failed. No retries permitted until 2026-09-10 17:11:38.01270397 +0000 UTC m=+3.993783952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmc4z" (UniqueName: "kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z") pod "network-check-target-t5lgb" (UID: "66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:37.012947 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.012821 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret podName:0f7c913b-fedc-4ca8-9d95-fa73663da5d4 nodeName:}" failed. No retries permitted until 2026-09-10 17:11:38.01278812 +0000 UTC m=+3.993868118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret") pod "global-pull-secret-syncer-8r6bk" (UID: "0f7c913b-fedc-4ca8-9d95-fa73663da5d4") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:37.113015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.112970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:37.113203 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.113109 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:37.113203 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:37.113189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:11:38.113167492 +0000 UTC m=+4.094247473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:37.440402 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.440276 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:06:36 +0000 UTC" deadline="2028-04-20 04:01:50.786634784 +0000 UTC" Sep 10 17:11:37.440402 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.440352 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14098h50m13.346287457s" Sep 10 17:11:37.495199 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.495159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d9247" event={"ID":"40439c75-55c7-42f5-9d84-6b380daf2e05","Type":"ContainerStarted","Data":"e1623e1d7a6e5d099bbfc393c532fa087ebc4bb4d4aa1ef047ba1479743e00cd"} Sep 10 17:11:37.498067 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.498032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqf2n" event={"ID":"6fbd2036-b001-419a-a567-ac76cd7a7802","Type":"ContainerStarted","Data":"da7ba14e69c6fb1628f7413e506de3fd063c2c1ffb0cbbacafc4f3ed68573abf"} Sep 10 17:11:37.500310 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.500225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lgfj2" event={"ID":"67b82be8-09d6-4631-bff0-14481c445847","Type":"ContainerStarted","Data":"a12d215dfea36ebbdd260a9c8a5db1480e3a32a4080f221d2822b8587e326656"} Sep 10 17:11:37.502118 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.501990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"f3f36fe73dab0c1352ed38e6fe4d70712de15bb997c936544e2acb882f141b49"} Sep 10 17:11:37.505580 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.505554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b4rjd" event={"ID":"af136ee8-5433-4faa-868a-83038e568060","Type":"ContainerStarted","Data":"f5737562b939d204837800101c9c44bb1b051773c4b779efd784fb86911d4247"} Sep 10 17:11:37.507538 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.507504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerStarted","Data":"5da507621ce8929350c36c0aaa2d1f916f5b0c671f967bb8801d3da8e184b60a"} Sep 10 17:11:37.509274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.509247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" event={"ID":"485fe935-b6ec-4702-87a5-a406108d9032","Type":"ContainerStarted","Data":"2c1b2a2a3759fba34759e766a513302ef28c94f319c03ebef8213c8684d6ab12"} Sep 10 17:11:37.511161 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.511106 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9khk5" event={"ID":"30c0389e-f4f6-45a9-b319-06639ac1a224","Type":"ContainerStarted","Data":"de5780611edea2689553be1b670caf3934907b99117c588da780cf39f99607ed"} Sep 10 17:11:37.513293 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:37.513266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d2rkj" event={"ID":"f6feefb4-04cc-497b-870f-609aab9359f9","Type":"ContainerStarted","Data":"a68df7e4b56fadb8fa518af033342d9db3a083d950a19ab612d0c094cb3b487e"} Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.021749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.021824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.021967 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.021971 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.022003 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.022019 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pmc4z for pod openshift-network-diagnostics/network-check-target-t5lgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.022037 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret podName:0f7c913b-fedc-4ca8-9d95-fa73663da5d4 nodeName:}" failed. No retries permitted until 2026-09-10 17:11:40.022016428 +0000 UTC m=+6.003096430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret") pod "global-pull-secret-syncer-8r6bk" (UID: "0f7c913b-fedc-4ca8-9d95-fa73663da5d4") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:38.022426 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.022074 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z podName:66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d nodeName:}" failed. No retries permitted until 2026-09-10 17:11:40.022056876 +0000 UTC m=+6.003136860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmc4z" (UniqueName: "kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z") pod "network-check-target-t5lgb" (UID: "66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:38.122713 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.122667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:38.122900 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.122852 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:38.123022 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.123007 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:11:40.122982152 +0000 UTC m=+6.104062133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:38.440686 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.440639 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:06:36 +0000 UTC" deadline="2028-03-17 13:34:54.665684021 +0000 UTC" Sep 10 17:11:38.440686 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.440685 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13292h23m16.225004866s" Sep 10 17:11:38.482907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.482085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:38.482907 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.482217 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:38.482907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.482651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:38.482907 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.482760 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:38.483274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:38.483058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:38.483274 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:38.483191 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:39.526011 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:39.525975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" event={"ID":"56c1d6772edd98b5770557256f012f4e","Type":"ContainerStarted","Data":"9e9dec13a1493ba1c1404bee4371ce375fd745b5f3aa6c0e929ed32d085bf5c0"} Sep 10 17:11:40.043512 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.043471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:40.043702 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.043572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:40.043766 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.043721 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:11:40.043766 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.043744 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:11:40.043766 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.043761 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pmc4z for pod openshift-network-diagnostics/network-check-target-t5lgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:40.043946 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.043823 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z podName:66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d nodeName:}" failed. No retries permitted until 2026-09-10 17:11:44.043803454 +0000 UTC m=+10.024883432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmc4z" (UniqueName: "kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z") pod "network-check-target-t5lgb" (UID: "66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:40.044275 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.044254 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:40.044369 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.044310 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret podName:0f7c913b-fedc-4ca8-9d95-fa73663da5d4 nodeName:}" failed. No retries permitted until 2026-09-10 17:11:44.044295019 +0000 UTC m=+10.025374997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret") pod "global-pull-secret-syncer-8r6bk" (UID: "0f7c913b-fedc-4ca8-9d95-fa73663da5d4") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:40.144447 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.144407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:40.144640 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.144564 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:40.144640 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.144635 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:11:44.144614112 +0000 UTC m=+10.125694092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:40.481060 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.481024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:40.481263 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.481167 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:40.481593 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.481574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:40.481749 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.481685 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:40.483064 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.483043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:40.483174 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:40.483149 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:40.542158 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.542116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" event={"ID":"f318e2a8b58477e08e99d42f801553c7","Type":"ContainerStarted","Data":"31f552004ec12ada61d016f77743bf2ee6e5539fe098afd978059af9afe7b7cd"} Sep 10 17:11:40.560420 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:40.560122 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-217.ec2.internal" podStartSLOduration=4.560097097 podStartE2EDuration="4.560097097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:39.543021963 +0000 UTC m=+5.524101966" watchObservedRunningTime="2026-09-10 17:11:40.560097097 +0000 UTC m=+6.541177100" Sep 10 17:11:42.481110 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:42.481072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:42.481110 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:42.481104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:42.481652 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:42.481083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:42.481652 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:42.481214 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:42.481652 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:42.481331 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:42.481652 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:42.481441 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:42.549632 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:42.549264 2576 generic.go:356] "Generic (PLEG): container finished" podID="f318e2a8b58477e08e99d42f801553c7" containerID="31f552004ec12ada61d016f77743bf2ee6e5539fe098afd978059af9afe7b7cd" exitCode=0 Sep 10 17:11:42.549632 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:42.549324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" event={"ID":"f318e2a8b58477e08e99d42f801553c7","Type":"ContainerDied","Data":"31f552004ec12ada61d016f77743bf2ee6e5539fe098afd978059af9afe7b7cd"} Sep 10 17:11:44.077890 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:44.077825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:44.078363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:44.077953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:44.078363 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.078095 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:44.078363 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.078103 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:11:44.078363 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.078180 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret podName:0f7c913b-fedc-4ca8-9d95-fa73663da5d4 nodeName:}" failed. No retries permitted until 2026-09-10 17:11:52.078158046 +0000 UTC m=+18.059238034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret") pod "global-pull-secret-syncer-8r6bk" (UID: "0f7c913b-fedc-4ca8-9d95-fa73663da5d4") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:44.078363 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.078183 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:11:44.078363 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.078198 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pmc4z for pod openshift-network-diagnostics/network-check-target-t5lgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:44.078363 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.078249 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z podName:66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d nodeName:}" failed. No retries permitted until 2026-09-10 17:11:52.078233387 +0000 UTC m=+18.059313370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmc4z" (UniqueName: "kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z") pod "network-check-target-t5lgb" (UID: "66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:44.179049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:44.179009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:44.179217 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.179168 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:44.179278 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.179244 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:11:52.179221792 +0000 UTC m=+18.160301771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:44.481334 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:44.481254 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:44.481488 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.481382 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:44.482030 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:44.481584 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:44.482030 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:44.481681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:44.482030 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.481781 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:44.482030 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:44.481859 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:46.480783 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:46.480752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:46.481292 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:46.480897 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:46.481292 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:46.480894 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:46.481292 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:46.480963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:46.481292 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:46.481069 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:46.481292 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:46.481135 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:48.480976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.480765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:48.481404 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:48.481080 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:48.481404 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.480802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:48.481404 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:48.481172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:48.481404 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.480765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:48.481404 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:48.481245 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:48.565994 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.565961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b4rjd" event={"ID":"af136ee8-5433-4faa-868a-83038e568060","Type":"ContainerStarted","Data":"9709030e3671447cc91599c17c6b41a25d7d8aece246e656bf8a07ae50ead404"} Sep 10 17:11:48.567802 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.567775 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c95e10-40c9-4da0-be08-744fb5e1cab8" containerID="7ddfefb9e28f2bae481b8da64d44b53b5008311df7d60fb0ab6cf096428a5ab4" exitCode=0 Sep 10 17:11:48.567965 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.567870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerDied","Data":"7ddfefb9e28f2bae481b8da64d44b53b5008311df7d60fb0ab6cf096428a5ab4"} Sep 10 17:11:48.570131 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.570104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" event={"ID":"485fe935-b6ec-4702-87a5-a406108d9032","Type":"ContainerStarted","Data":"90bac2cdeee8479f62cafdfdb8057f1045527d641907b1b0d851f23d0956c9b6"} Sep 10 17:11:48.571934 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.571913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9khk5" event={"ID":"30c0389e-f4f6-45a9-b319-06639ac1a224","Type":"ContainerStarted","Data":"ef49a198feeafee46d483c2c00860cc9482799693684b97501655242a5e77b41"} Sep 10 17:11:48.573541 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.573519 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d2rkj" event={"ID":"f6feefb4-04cc-497b-870f-609aab9359f9","Type":"ContainerStarted","Data":"f7f351bea34356883570acd190e732d63f8ca6c9f7dcaf8f0fc165d99822d126"} Sep 10 17:11:48.575695 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.575663 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" event={"ID":"f318e2a8b58477e08e99d42f801553c7","Type":"ContainerStarted","Data":"5bd0084b0d32b96f2325bfdaeff0b363c023923972a355f80c296c20635ad37f"} Sep 10 17:11:48.577269 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.577245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d9247" event={"ID":"40439c75-55c7-42f5-9d84-6b380daf2e05","Type":"ContainerStarted","Data":"0141c8ea59aecd4b8fc4dceff10aecf73ab406435a06e25fd86bd509f96b4d78"} Sep 10 17:11:48.581693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.581655 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b4rjd" podStartSLOduration=2.817551316 podStartE2EDuration="13.581642874s" totalImagesPullingTime="10.764091558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:48.581078797 +0000 UTC m=+14.562158794" watchObservedRunningTime="2026-09-10 17:11:48.581642874 +0000 UTC m=+14.562722871" Sep 10 17:11:48.613024 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.612966 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-217.ec2.internal" podStartSLOduration=12.612951025 podStartE2EDuration="12.612951025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:48.61257627 +0000 UTC m=+14.593656268" watchObservedRunningTime="2026-09-10 17:11:48.612951025 +0000 UTC m=+14.594031002" Sep 10 17:11:48.613354 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.613313 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d9247" podStartSLOduration=2.9016075150000002 podStartE2EDuration="13.613300781s" totalImagesPullingTime="10.711693266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:48.595630664 +0000 UTC m=+14.576710673" watchObservedRunningTime="2026-09-10 17:11:48.613300781 +0000 UTC m=+14.594380787" Sep 10 17:11:48.626622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.626561 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d2rkj" podStartSLOduration=2.848686441 podStartE2EDuration="13.626547407s" totalImagesPullingTime="10.777860966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:48.626385464 +0000 UTC m=+14.607465462" watchObservedRunningTime="2026-09-10 17:11:48.626547407 +0000 UTC m=+14.607627406" Sep 10 17:11:48.643633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:48.643576 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9khk5" podStartSLOduration=2.589367254 podStartE2EDuration="13.643559379s" totalImagesPullingTime="11.054192125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:48.643348939 +0000 UTC m=+14.624428938" watchObservedRunningTime="2026-09-10 17:11:48.643559379 +0000 UTC m=+14.624639378" Sep 10 17:11:49.572119 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:49.572079 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:49.572809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:49.572788 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:49.581430 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:49.581367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqf2n" event={"ID":"6fbd2036-b001-419a-a567-ac76cd7a7802","Type":"ContainerStarted","Data":"3796c4bd949131217bea2c913f6cd0c495284f00f35974df3ca186c99573603c"} Sep 10 17:11:49.604555 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:49.604474 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xqf2n" podStartSLOduration=3.8858101290000002 podStartE2EDuration="14.604452453s" totalImagesPullingTime="10.718642324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:49.603692138 +0000 UTC m=+15.584772137" watchObservedRunningTime="2026-09-10 17:11:49.604452453 +0000 UTC m=+15.585532451" Sep 10 17:11:50.480248 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:50.480207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:50.480422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:50.480207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:50.480422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:50.480347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:50.480422 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:50.480342 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:50.480598 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:50.480464 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:50.480598 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:50.480561 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:50.583572 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:50.583537 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 17:11:51.985895 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:51.985843 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:51.986422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:51.986000 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 17:11:51.986607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:51.986586 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b4rjd" Sep 10 17:11:52.139917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:52.139864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:52.140242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:52.139958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:52.140242 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.140052 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:11:52.140242 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.140075 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:11:52.140242 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.140089 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pmc4z for pod openshift-network-diagnostics/network-check-target-t5lgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:52.140242 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.140103 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:52.140242 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.140176 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret podName:0f7c913b-fedc-4ca8-9d95-fa73663da5d4 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:08.140156042 +0000 UTC m=+34.121236036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret") pod "global-pull-secret-syncer-8r6bk" (UID: "0f7c913b-fedc-4ca8-9d95-fa73663da5d4") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:11:52.140242 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.140196 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z podName:66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d nodeName:}" failed. No retries permitted until 2026-09-10 17:12:08.140186547 +0000 UTC m=+34.121266526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmc4z" (UniqueName: "kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z") pod "network-check-target-t5lgb" (UID: "66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:11:52.240617 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:52.240526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:52.240771 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.240683 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:52.240771 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.240751 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:12:08.240734859 +0000 UTC m=+34.221814842 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:11:52.480512 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:52.480473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:52.480693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:52.480521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:52.480693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:52.480473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:52.480693 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.480609 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:52.480693 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.480687 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:52.480976 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:52.480791 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:54.481378 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:54.481184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:54.481835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:54.481264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:54.481835 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:54.481465 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:54.481835 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:54.481570 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:54.481835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:54.481288 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:54.481835 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:54.481663 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:55.595897 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:55.595831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"4059003ffb812197bc12ac452e7764bf97396ec2324cabba395b9cea57437e2d"} Sep 10 17:11:55.631232 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:55.631021 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 17:11:56.451957 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.451650 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T17:11:55.631041231Z","UUID":"7dbe7a22-fd18-4f5c-84e1-bdc64477a121","Handler":null,"Name":"","Endpoint":""} Sep 10 17:11:56.453570 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.453546 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 17:11:56.453710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.453580 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 17:11:56.480710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.480678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:56.480894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.480678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:56.480894 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:56.480804 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:56.480894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.480812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:56.481030 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:56.480931 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:56.481030 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:56.481013 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:56.602684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.602647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lgfj2" event={"ID":"67b82be8-09d6-4631-bff0-14481c445847","Type":"ContainerStarted","Data":"4b983a50a463e1e4107c5fd0ff97f4aa8b038f220b04ea590047d314d345f5ea"} Sep 10 17:11:56.606986 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.606912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"2899ce5167259b699931f049918d9449ba4fec63c6f7bffb3f1c0c9629963f55"} Sep 10 17:11:56.606986 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.606992 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"4a746e4fd9fe1ff710359496ad3777f4cd0cf6a510a69ea88274fc11c162ccbf"} Sep 10 17:11:56.607227 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.607040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"4b9a9e3112e359ee0a934d2a3b5dc65954532f739ac6a8e92ca0ad574f53bb2c"} Sep 10 17:11:56.607227 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.607088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"63030d63329d1bd5f49ded6528f1ae9eb6278a2761ddd4b84491063f62610237"} Sep 10 17:11:56.607227 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.607138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"fcf41d69d784de9d21e84d85efb1e56ae626c7882d013025897219b3e462a92b"} Sep 10 17:11:56.608986 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.608959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" event={"ID":"485fe935-b6ec-4702-87a5-a406108d9032","Type":"ContainerStarted","Data":"1b89def69c58f199a18471441f570c64815727b1adcb8f779b6c466d68e63aeb"} Sep 10 17:11:56.621265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:56.621221 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lgfj2" podStartSLOduration=2.853034833 podStartE2EDuration="21.621207567s" totalImagesPullingTime="18.768172734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:56.620800544 +0000 UTC m=+22.601880548" watchObservedRunningTime="2026-09-10 17:11:56.621207567 +0000 UTC m=+22.602287564" Sep 10 17:11:57.614453 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:57.614405 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" event={"ID":"485fe935-b6ec-4702-87a5-a406108d9032","Type":"ContainerStarted","Data":"ebb951ce71ff7fd7b680d59ee6cf5bbbfb029657e8d238575aea0d6d48e0befe"} Sep 10 17:11:57.632616 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:57.632568 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9q9fb" podStartSLOduration=3.108265087 podStartE2EDuration="22.632549385s" totalImagesPullingTime="19.524284298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:57.632467949 +0000 UTC m=+23.613547948" watchObservedRunningTime="2026-09-10 17:11:57.632549385 +0000 UTC m=+23.613629384" Sep 10 17:11:58.480349 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:58.480313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:11:58.480520 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:58.480314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:11:58.480520 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:58.480422 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:11:58.480609 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:58.480529 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:11:58.480609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:58.480313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:11:58.480675 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:11:58.480640 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:11:58.621216 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:11:58.621184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"c6ddf34e35407fe595b3935fd938cd5ccc266393c47276b9c34500c21716b36d"} Sep 10 17:12:00.480630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:00.480546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:00.480630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:00.480586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:00.481138 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:00.480545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:00.481138 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:00.480677 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:12:00.481138 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:00.480781 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:12:00.481138 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:00.480874 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:12:01.632888 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:01.632838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" event={"ID":"ea205362-1082-4392-8909-ee82fc215754","Type":"ContainerStarted","Data":"6798cc0a487834ca3783d775fa6b7271fff00eb81aaa8ccd97c84de7da9a518b"} Sep 10 17:12:01.633352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:01.633120 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:12:01.633352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:01.633195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:12:01.634943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:01.634911 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c95e10-40c9-4da0-be08-744fb5e1cab8" containerID="1aea42213a49c3516d0d57b1d0c6bff07b70b026f2ae2e145c5fe897da46afe5" exitCode=0 Sep 10 17:12:01.635069 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:01.634967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerDied","Data":"1aea42213a49c3516d0d57b1d0c6bff07b70b026f2ae2e145c5fe897da46afe5"} Sep 10 17:12:01.648799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:01.648779 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:12:01.660398 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:01.660356 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" podStartSLOduration=7.923472965 podStartE2EDuration="26.660342948s" totalImagesPullingTime="18.736869983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:01.659013536 +0000 UTC m=+27.640093534" watchObservedRunningTime="2026-09-10 17:12:01.660342948 +0000 UTC m=+27.641422945" Sep 10 17:12:02.481025 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.480990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:02.481188 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.480991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:02.481188 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:02.481132 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:12:02.481188 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.480991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:02.481188 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:02.481175 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:12:02.481392 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:02.481262 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:12:02.638672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.638562 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:12:02.659047 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.658733 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:12:02.924191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.924154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t5lgb"] Sep 10 17:12:02.924338 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.924318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:02.924446 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:02.924423 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:12:02.926833 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.926806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8r6bk"] Sep 10 17:12:02.926978 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.926925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:02.927041 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:02.927021 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:12:02.936022 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.935999 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-555sb"] Sep 10 17:12:02.936130 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:02.936091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:02.936193 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:02.936177 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:12:03.642085 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:03.642047 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c95e10-40c9-4da0-be08-744fb5e1cab8" containerID="416822315ca6908c3a5295ece41b0fbfe4a041ebf22bceecc177b751c279e85b" exitCode=0 Sep 10 17:12:03.642485 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:03.642131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerDied","Data":"416822315ca6908c3a5295ece41b0fbfe4a041ebf22bceecc177b751c279e85b"} Sep 10 17:12:04.480762 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:04.480738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:04.480897 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:04.480820 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:12:04.480897 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:04.480861 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:04.481034 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:04.481007 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:12:04.481088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:04.481025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:04.481178 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:04.481156 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:12:04.646425 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:04.646391 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c95e10-40c9-4da0-be08-744fb5e1cab8" containerID="84db70ebc7d3fe48b104c04a58e3f99a46e1422e94fa0bc3e31153415d709a70" exitCode=0 Sep 10 17:12:04.647034 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:04.646472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerDied","Data":"84db70ebc7d3fe48b104c04a58e3f99a46e1422e94fa0bc3e31153415d709a70"} Sep 10 17:12:06.481210 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:06.481030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:06.481778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:06.481030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:06.481778 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:06.481284 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6bk" podUID="0f7c913b-fedc-4ca8-9d95-fa73663da5d4" Sep 10 17:12:06.481778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:06.481111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:06.481778 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:06.481367 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-555sb" podUID="4721421f-ab96-45e5-8503-ccc0a9a3f83f" Sep 10 17:12:06.481778 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:06.481457 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t5lgb" podUID="66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d" Sep 10 17:12:08.155488 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.155451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:08.155875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.155502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:08.155875 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.155607 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:12:08.155875 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.155622 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:12:08.155875 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.155642 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:12:08.155875 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.155655 2576 projected.go:196] Error preparing data for projected volume kube-api-access-pmc4z for pod openshift-network-diagnostics/network-check-target-t5lgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:12:08.155875 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.155673 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret podName:0f7c913b-fedc-4ca8-9d95-fa73663da5d4 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:40.155655391 +0000 UTC m=+66.136735384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret") pod "global-pull-secret-syncer-8r6bk" (UID: "0f7c913b-fedc-4ca8-9d95-fa73663da5d4") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:12:08.155875 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.155690 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z podName:66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d nodeName:}" failed. No retries permitted until 2026-09-10 17:12:40.155680015 +0000 UTC m=+66.136759997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmc4z" (UniqueName: "kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z") pod "network-check-target-t5lgb" (UID: "66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:12:08.256171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.256065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:08.256324 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.256218 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:12:08.256324 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.256300 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:12:40.256279747 +0000 UTC m=+66.237359726 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:12:08.285171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.285140 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-217.ec2.internal" event="NodeReady" Sep 10 17:12:08.285319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.285287 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 17:12:08.320958 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.319464 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-849f8f8ffd-n4xtz"] Sep 10 17:12:08.407990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.407957 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xbd5r"] Sep 10 17:12:08.408151 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.408079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.410641 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.410617 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 17:12:08.410892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.410867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ppxr2\"" Sep 10 17:12:08.410984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.410867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 17:12:08.410984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.410867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 17:12:08.415673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.415646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 17:12:08.433949 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.433925 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849f8f8ffd-n4xtz"] Sep 10 17:12:08.434070 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.433964 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cndb6"] Sep 10 17:12:08.434070 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.434040 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.436468 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.436449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 17:12:08.436587 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.436505 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 17:12:08.436587 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.436519 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52r9p\"" Sep 10 17:12:08.457056 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.457031 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cndb6"] Sep 10 17:12:08.457186 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.457064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbd5r"] Sep 10 17:12:08.457186 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.457114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:08.462918 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.462643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 17:12:08.462918 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.462728 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 17:12:08.462918 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.462729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 17:12:08.463861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.463822 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gj68b\"" Sep 10 17:12:08.480782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.480753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:08.480782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.480770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:08.481282 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.481246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:08.484474 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.484447 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 17:12:08.484570 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.484500 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vgb7\"" Sep 10 17:12:08.486284 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.486261 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 17:12:08.486389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.486349 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2g8g\"" Sep 10 17:12:08.489516 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.489459 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 17:12:08.489761 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.489743 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 17:12:08.558511 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fd67450-0bb9-4894-8e12-02053f31faf1-ca-trust-extracted\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558511 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-image-registry-private-configuration\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cc5e36-0e94-4097-b240-bd5660695b52-config-volume\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.558693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhzz\" (UniqueName: \"kubernetes.io/projected/a9cc5e36-0e94-4097-b240-bd5660695b52-kube-api-access-7hhzz\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.558693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.558693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-installation-pull-secrets\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cc5e36-0e94-4097-b240-bd5660695b52-tmp-dir\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.558693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-certificates\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-bound-sa-token\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7428d8c1-26fb-4a73-b4c2-ff5cee1533cc-cert\") pod \"ingress-canary-cndb6\" (UID: \"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc\") " pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:08.558927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-trusted-ca\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxq2\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-kube-api-access-7mxq2\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.558927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.558835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzfz\" (UniqueName: \"kubernetes.io/projected/7428d8c1-26fb-4a73-b4c2-ff5cee1533cc-kube-api-access-7qzfz\") pod \"ingress-canary-cndb6\" (UID: \"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc\") " pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:08.659727 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-bound-sa-token\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.659727 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7428d8c1-26fb-4a73-b4c2-ff5cee1533cc-cert\") pod \"ingress-canary-cndb6\" (UID: \"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc\") " pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-trusted-ca\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxq2\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-kube-api-access-7mxq2\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzfz\" (UniqueName: \"kubernetes.io/projected/7428d8c1-26fb-4a73-b4c2-ff5cee1533cc-kube-api-access-7qzfz\") pod \"ingress-canary-cndb6\" (UID: \"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc\") " pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fd67450-0bb9-4894-8e12-02053f31faf1-ca-trust-extracted\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-image-registry-private-configuration\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cc5e36-0e94-4097-b240-bd5660695b52-config-volume\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhzz\" (UniqueName: \"kubernetes.io/projected/a9cc5e36-0e94-4097-b240-bd5660695b52-kube-api-access-7hhzz\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.659983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.659983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.660360 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.660007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-installation-pull-secrets\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.660360 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.660021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cc5e36-0e94-4097-b240-bd5660695b52-tmp-dir\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.660360 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.660066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-certificates\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.660655 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.660630 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:12:08.660778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.660682 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-certificates\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.660778 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.660708 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls podName:a9cc5e36-0e94-4097-b240-bd5660695b52 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:09.160688756 +0000 UTC m=+35.141768742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls") pod "dns-default-xbd5r" (UID: "a9cc5e36-0e94-4097-b240-bd5660695b52") : secret "dns-default-metrics-tls" not found Sep 10 17:12:08.661368 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.661151 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 17:12:08.661368 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.661175 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-849f8f8ffd-n4xtz: secret "image-registry-tls" not found Sep 10 17:12:08.661368 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:08.661221 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls podName:3fd67450-0bb9-4894-8e12-02053f31faf1 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:09.161204451 +0000 UTC m=+35.142284427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls") pod "image-registry-849f8f8ffd-n4xtz" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1") : secret "image-registry-tls" not found Sep 10 17:12:08.661368 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.661273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fd67450-0bb9-4894-8e12-02053f31faf1-ca-trust-extracted\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.661579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.661453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-trusted-ca\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.661626 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.661607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cc5e36-0e94-4097-b240-bd5660695b52-tmp-dir\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.661920 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.661896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cc5e36-0e94-4097-b240-bd5660695b52-config-volume\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.664254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.664232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-installation-pull-secrets\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.664341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.664248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-image-registry-private-configuration\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.664341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.664269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7428d8c1-26fb-4a73-b4c2-ff5cee1533cc-cert\") pod \"ingress-canary-cndb6\" (UID: \"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc\") " pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:08.670011 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.669962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-bound-sa-token\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.670364 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.670200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxq2\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-kube-api-access-7mxq2\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:08.670364 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.670238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzfz\" (UniqueName: \"kubernetes.io/projected/7428d8c1-26fb-4a73-b4c2-ff5cee1533cc-kube-api-access-7qzfz\") pod \"ingress-canary-cndb6\" (UID: \"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc\") " pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:08.670364 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.670355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhzz\" (UniqueName: \"kubernetes.io/projected/a9cc5e36-0e94-4097-b240-bd5660695b52-kube-api-access-7hhzz\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:08.766383 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:08.766347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cndb6" Sep 10 17:12:09.091578 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:09.091544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cndb6"] Sep 10 17:12:09.164792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:09.164703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:09.164792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:09.164757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:09.165291 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:09.164913 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:12:09.165291 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:09.164914 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 17:12:09.165291 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:09.164940 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-849f8f8ffd-n4xtz: secret "image-registry-tls" not found Sep 10 17:12:09.165291 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:09.164994 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls podName:a9cc5e36-0e94-4097-b240-bd5660695b52 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:10.164975461 +0000 UTC m=+36.146055438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls") pod "dns-default-xbd5r" (UID: "a9cc5e36-0e94-4097-b240-bd5660695b52") : secret "dns-default-metrics-tls" not found Sep 10 17:12:09.165291 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:09.165009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls podName:3fd67450-0bb9-4894-8e12-02053f31faf1 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:10.165003501 +0000 UTC m=+36.146083476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls") pod "image-registry-849f8f8ffd-n4xtz" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1") : secret "image-registry-tls" not found Sep 10 17:12:10.172174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:10.172137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:10.172174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:10.172182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:10.172738 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:10.172328 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 17:12:10.172738 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:10.172352 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-849f8f8ffd-n4xtz: secret "image-registry-tls" not found Sep 10 17:12:10.172738 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:10.172413 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:12:10.172738 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:10.172417 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls podName:3fd67450-0bb9-4894-8e12-02053f31faf1 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:12.172401162 +0000 UTC m=+38.153481138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls") pod "image-registry-849f8f8ffd-n4xtz" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1") : secret "image-registry-tls" not found Sep 10 17:12:10.172738 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:10.172504 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls podName:a9cc5e36-0e94-4097-b240-bd5660695b52 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:12.172483584 +0000 UTC m=+38.153563566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls") pod "dns-default-xbd5r" (UID: "a9cc5e36-0e94-4097-b240-bd5660695b52") : secret "dns-default-metrics-tls" not found Sep 10 17:12:10.651640 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:10.651605 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7428d8c1_26fb_4a73_b4c2_ff5cee1533cc.slice/crio-ab8535a93645818b7534704b43b621d23e8f0009844441127a9a85a52ba81b34 WatchSource:0}: Error finding container ab8535a93645818b7534704b43b621d23e8f0009844441127a9a85a52ba81b34: Status 404 returned error can't find the container with id ab8535a93645818b7534704b43b621d23e8f0009844441127a9a85a52ba81b34 Sep 10 17:12:10.661036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:10.660869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cndb6" event={"ID":"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc","Type":"ContainerStarted","Data":"ab8535a93645818b7534704b43b621d23e8f0009844441127a9a85a52ba81b34"} Sep 10 17:12:11.667401 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:11.667362 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c95e10-40c9-4da0-be08-744fb5e1cab8" containerID="cb8f40e5c876812b2d1e21f8d3945476ce3132cd19ec35b481a2a85800eb613a" exitCode=0 Sep 10 17:12:11.667817 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:11.667408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerDied","Data":"cb8f40e5c876812b2d1e21f8d3945476ce3132cd19ec35b481a2a85800eb613a"} Sep 10 17:12:12.192428 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:12.192393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:12.192620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:12.192496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:12.192620 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:12.192558 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:12:12.192620 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:12.192594 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 17:12:12.192620 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:12.192613 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-849f8f8ffd-n4xtz: secret "image-registry-tls" not found Sep 10 17:12:12.192818 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:12.192629 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls podName:a9cc5e36-0e94-4097-b240-bd5660695b52 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:16.19260939 +0000 UTC m=+42.173689379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls") pod "dns-default-xbd5r" (UID: "a9cc5e36-0e94-4097-b240-bd5660695b52") : secret "dns-default-metrics-tls" not found Sep 10 17:12:12.192818 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:12.192667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls podName:3fd67450-0bb9-4894-8e12-02053f31faf1 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:16.192649764 +0000 UTC m=+42.173729746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls") pod "image-registry-849f8f8ffd-n4xtz" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1") : secret "image-registry-tls" not found Sep 10 17:12:12.672524 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:12.672484 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c95e10-40c9-4da0-be08-744fb5e1cab8" containerID="ed15c2d42a4bbb458914e957659d5a0f89be74ef199b756d01222a3448cb8f18" exitCode=0 Sep 10 17:12:12.672974 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:12.672568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerDied","Data":"ed15c2d42a4bbb458914e957659d5a0f89be74ef199b756d01222a3448cb8f18"} Sep 10 17:12:13.677647 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:13.677617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r256" event={"ID":"a2c95e10-40c9-4da0-be08-744fb5e1cab8","Type":"ContainerStarted","Data":"96456023b23856cec4dc63e10c5074694bbf7968ae9d30ea9c3f8ea068d569e9"} Sep 10 17:12:13.706131 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:13.706036 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9r256" podStartSLOduration=7.163720292 podStartE2EDuration="38.706020695s" totalImagesPullingTime="31.542300403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:13.70380178 +0000 UTC m=+39.684881768" watchObservedRunningTime="2026-09-10 17:12:13.706020695 +0000 UTC m=+39.687100693" Sep 10 17:12:13.914748 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:13.914713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:14.002651 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:14.002561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:14.681927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:14.681875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cndb6" event={"ID":"7428d8c1-26fb-4a73-b4c2-ff5cee1533cc","Type":"ContainerStarted","Data":"ff603185d09480966740e9c8fb1fbe62453c61102d2d6c84a02fa0abf36c8c82"} Sep 10 17:12:14.697815 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:14.697751 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cndb6" podStartSLOduration=3.5501143600000002 podStartE2EDuration="6.697735367s" totalImagesPullingTime="3.147621007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:14.696897904 +0000 UTC m=+40.677977896" watchObservedRunningTime="2026-09-10 17:12:14.697735367 +0000 UTC m=+40.678815343" Sep 10 17:12:16.220087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:16.220051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:16.220087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:16.220090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:16.220488 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:16.220193 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:12:16.220488 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:16.220213 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 17:12:16.220488 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:16.220234 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-849f8f8ffd-n4xtz: secret "image-registry-tls" not found Sep 10 17:12:16.220488 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:16.220258 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls podName:a9cc5e36-0e94-4097-b240-bd5660695b52 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:24.220245429 +0000 UTC m=+50.201325405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls") pod "dns-default-xbd5r" (UID: "a9cc5e36-0e94-4097-b240-bd5660695b52") : secret "dns-default-metrics-tls" not found Sep 10 17:12:16.220488 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:16.220287 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls podName:3fd67450-0bb9-4894-8e12-02053f31faf1 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:24.220273886 +0000 UTC m=+50.201353862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls") pod "image-registry-849f8f8ffd-n4xtz" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1") : secret "image-registry-tls" not found Sep 10 17:12:16.670133 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:16.670099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:16.770753 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:16.770714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:17.624347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:17.624312 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:17.725766 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:17.725728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:18.633500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:18.633470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:18.732957 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:18.732920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:19.580193 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:19.580140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:19.635817 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:19.635778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:21.520841 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:21.520665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:21.576543 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:21.576512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:22.529319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:22.529285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:23.896369 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.896335 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s"] Sep 10 17:12:23.921479 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.921448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68d5b75dfd-l6m8l"] Sep 10 17:12:23.921621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.921564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:23.924245 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.924222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 17:12:23.925658 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.925641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bqrg9\"" Sep 10 17:12:23.925756 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.925721 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 17:12:23.925811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.925728 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 17:12:23.926790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.926774 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 17:12:23.933109 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.933087 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s"] Sep 10 17:12:23.933193 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.933121 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d5b75dfd-l6m8l"] Sep 10 17:12:23.933237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.933199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:23.936435 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.936418 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 17:12:23.936532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.936465 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 17:12:23.936591 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.936422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 17:12:23.936643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.936420 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 17:12:23.936762 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.936745 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 17:12:23.936835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.936779 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 17:12:23.936835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:23.936820 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-b9bhx\"" Sep 10 17:12:24.010408 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.010371 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj"] Sep 10 17:12:24.020715 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.020692 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz"] Sep 10 17:12:24.020846 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.020811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" Sep 10 17:12:24.023661 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.023633 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 17:12:24.023661 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.023655 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:12:24.028067 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.028046 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fkcnp\"" Sep 10 17:12:24.032940 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.032922 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-tv8w9"] Sep 10 17:12:24.033033 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.033020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:24.035590 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.035573 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 17:12:24.035870 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.035851 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:12:24.036429 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.036411 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9qcbx\"" Sep 10 17:12:24.037144 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.037124 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 17:12:24.049133 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.049111 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j"] Sep 10 17:12:24.049248 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.049234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.051573 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.051556 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 17:12:24.051673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.051599 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-c66c2\"" Sep 10 17:12:24.051760 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.051746 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 17:12:24.052171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.052155 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 17:12:24.052448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.052436 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:12:24.066671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.066633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj"] Sep 10 17:12:24.066777 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.066682 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-tv8w9"] Sep 10 17:12:24.066820 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.066789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz"] Sep 10 17:12:24.066820 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.066815 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j"] Sep 10 17:12:24.066908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.066790 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.067513 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.067495 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 17:12:24.069566 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.069423 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 17:12:24.069699 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.069683 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-g8fg9\"" Sep 10 17:12:24.069761 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.069705 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 17:12:24.070045 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.070029 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:12:24.070993 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.070971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 17:12:24.072038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.072014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:24.076560 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhn28\" (UniqueName: \"kubernetes.io/projected/170c6e7a-0aa9-41f8-a475-904a458b67a0-kube-api-access-qhn28\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.076662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.076662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvpw\" (UniqueName: \"kubernetes.io/projected/99a001fe-c41e-457b-b8ff-64fa4da2af08-kube-api-access-4mvpw\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.076662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.076773 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/170c6e7a-0aa9-41f8-a475-904a458b67a0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.076773 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-stats-auth\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.076773 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-default-certificate\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.076773 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.076763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.124741 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.124708 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9"] Sep 10 17:12:24.134508 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.134482 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-wpxtc"] Sep 10 17:12:24.134619 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.134610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.137969 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.137930 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 17:12:24.138110 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.138002 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 17:12:24.138110 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.138016 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:12:24.138277 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.138264 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ssz2n\"" Sep 10 17:12:24.138354 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.138329 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 17:12:24.142975 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.142957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.146335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.146311 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 17:12:24.146483 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.146339 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jf75l\"" Sep 10 17:12:24.146556 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.146530 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 17:12:24.146615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.146593 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 17:12:24.147061 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.147048 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 17:12:24.155396 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.155374 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9"] Sep 10 17:12:24.155592 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.155575 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 17:12:24.160080 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.160061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-wpxtc"] Sep 10 17:12:24.177294 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthdq\" (UniqueName: \"kubernetes.io/projected/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-kube-api-access-vthdq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.177432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71a87714-cd43-4cd4-b7a6-d23ad894d1df-trusted-ca\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.177432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.177432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94shp\" (UniqueName: \"kubernetes.io/projected/71a87714-cd43-4cd4-b7a6-d23ad894d1df-kube-api-access-94shp\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.177432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71a87714-cd43-4cd4-b7a6-d23ad894d1df-serving-cert\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.177618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.177618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvpw\" (UniqueName: \"kubernetes.io/projected/99a001fe-c41e-457b-b8ff-64fa4da2af08-kube-api-access-4mvpw\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.177618 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.177516 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:12:24.177618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-default-certificate\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.177618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7dwk\" (UniqueName: \"kubernetes.io/projected/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-kube-api-access-s7dwk\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:24.177618 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.177589 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:24.677575862 +0000 UTC m=+50.658655837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : secret "router-metrics-certs-default" not found Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/170c6e7a-0aa9-41f8-a475-904a458b67a0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-stats-auth\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.177809 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.177859 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls podName:170c6e7a-0aa9-41f8-a475-904a458b67a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:24.677843546 +0000 UTC m=+50.658923527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6zv7s" (UID: "170c6e7a-0aa9-41f8-a475-904a458b67a0") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:24.177929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7j9\" (UniqueName: \"kubernetes.io/projected/5fa8506b-482d-456a-bab2-a7c26b3d8e64-kube-api-access-wl7j9\") pod \"volume-data-source-validator-695c895bd6-l2vqj\" (UID: \"5fa8506b-482d-456a-bab2-a7c26b3d8e64\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" Sep 10 17:12:24.178256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.177951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhn28\" (UniqueName: \"kubernetes.io/projected/170c6e7a-0aa9-41f8-a475-904a458b67a0-kube-api-access-qhn28\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.178256 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.177959 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:24.677947385 +0000 UTC m=+50.659027385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : configmap references non-existent config key: service-ca.crt Sep 10 17:12:24.178256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.178008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71a87714-cd43-4cd4-b7a6-d23ad894d1df-config\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.178478 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.178460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/170c6e7a-0aa9-41f8-a475-904a458b67a0-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.181252 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.181194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-default-certificate\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.181252 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.181214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-stats-auth\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.187586 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.187537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:24.189367 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.189342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvpw\" (UniqueName: \"kubernetes.io/projected/99a001fe-c41e-457b-b8ff-64fa4da2af08-kube-api-access-4mvpw\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.189668 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.189649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhn28\" (UniqueName: \"kubernetes.io/projected/170c6e7a-0aa9-41f8-a475-904a458b67a0-kube-api-access-qhn28\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.211828 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.211805 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pxdwb"] Sep 10 17:12:24.224796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.224772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" Sep 10 17:12:24.225552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.225526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pxdwb"] Sep 10 17:12:24.228184 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.228166 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-d6jxn\"" Sep 10 17:12:24.278640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.278608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-serving-cert\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.278831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.278667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:24.278831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.278718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7dwk\" (UniqueName: \"kubernetes.io/projected/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-kube-api-access-s7dwk\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:24.278831 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.278747 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:12:24.278831 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.278792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls podName:a9cc5e36-0e94-4097-b240-bd5660695b52 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:40.278778735 +0000 UTC m=+66.259858712 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls") pod "dns-default-xbd5r" (UID: "a9cc5e36-0e94-4097-b240-bd5660695b52") : secret "dns-default-metrics-tls" not found Sep 10 17:12:24.278831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.278748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-config\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.278872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8f7g\" (UniqueName: \"kubernetes.io/projected/77c8c142-262a-4f8e-aac2-fa61beaee390-kube-api-access-v8f7g\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.278959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.278976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk429\" (UniqueName: \"kubernetes.io/projected/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-kube-api-access-tk429\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c8c142-262a-4f8e-aac2-fa61beaee390-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.279031 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.279073 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls podName:5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:24.779061661 +0000 UTC m=+50.760141638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hz8hz" (UID: "5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9") : secret "samples-operator-tls" not found Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7j9\" (UniqueName: \"kubernetes.io/projected/5fa8506b-482d-456a-bab2-a7c26b3d8e64-kube-api-access-wl7j9\") pod \"volume-data-source-validator-695c895bd6-l2vqj\" (UID: \"5fa8506b-482d-456a-bab2-a7c26b3d8e64\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77c8c142-262a-4f8e-aac2-fa61beaee390-serving-cert\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.279146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71a87714-cd43-4cd4-b7a6-d23ad894d1df-config\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthdq\" (UniqueName: \"kubernetes.io/projected/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-kube-api-access-vthdq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77c8c142-262a-4f8e-aac2-fa61beaee390-tmp\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/77c8c142-262a-4f8e-aac2-fa61beaee390-snapshots\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71a87714-cd43-4cd4-b7a6-d23ad894d1df-trusted-ca\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c8c142-262a-4f8e-aac2-fa61beaee390-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.279335 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.279353 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-849f8f8ffd-n4xtz: secret "image-registry-tls" not found Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94shp\" (UniqueName: \"kubernetes.io/projected/71a87714-cd43-4cd4-b7a6-d23ad894d1df-kube-api-access-94shp\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.279399 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls podName:3fd67450-0bb9-4894-8e12-02053f31faf1 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:40.279381829 +0000 UTC m=+66.260461822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls") pod "image-registry-849f8f8ffd-n4xtz" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1") : secret "image-registry-tls" not found Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71a87714-cd43-4cd4-b7a6-d23ad894d1df-serving-cert\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.279685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.280166 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.279817 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71a87714-cd43-4cd4-b7a6-d23ad894d1df-config\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.280269 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.280251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71a87714-cd43-4cd4-b7a6-d23ad894d1df-trusted-ca\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.281786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.281757 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.281896 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.281851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71a87714-cd43-4cd4-b7a6-d23ad894d1df-serving-cert\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.286692 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.286668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7dwk\" (UniqueName: \"kubernetes.io/projected/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-kube-api-access-s7dwk\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:24.288190 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.288166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94shp\" (UniqueName: \"kubernetes.io/projected/71a87714-cd43-4cd4-b7a6-d23ad894d1df-kube-api-access-94shp\") pod \"console-operator-85b454874b-tv8w9\" (UID: \"71a87714-cd43-4cd4-b7a6-d23ad894d1df\") " pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.288272 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.288226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7j9\" (UniqueName: \"kubernetes.io/projected/5fa8506b-482d-456a-bab2-a7c26b3d8e64-kube-api-access-wl7j9\") pod \"volume-data-source-validator-695c895bd6-l2vqj\" (UID: \"5fa8506b-482d-456a-bab2-a7c26b3d8e64\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" Sep 10 17:12:24.288679 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.288661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthdq\" (UniqueName: \"kubernetes.io/projected/1a1aa361-e7eb-48f9-82e0-6bb7206e0beb-kube-api-access-vthdq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fxk8j\" (UID: \"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.330003 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.329972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" Sep 10 17:12:24.359385 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.359364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:24.375588 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.375560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" Sep 10 17:12:24.380204 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-serving-cert\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.380266 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9crp\" (UniqueName: \"kubernetes.io/projected/43627182-feb6-4df9-b293-93645921582f-kube-api-access-r9crp\") pod \"network-check-source-685456655f-pxdwb\" (UID: \"43627182-feb6-4df9-b293-93645921582f\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" Sep 10 17:12:24.380303 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-config\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.380346 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8f7g\" (UniqueName: \"kubernetes.io/projected/77c8c142-262a-4f8e-aac2-fa61beaee390-kube-api-access-v8f7g\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.380433 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk429\" (UniqueName: \"kubernetes.io/projected/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-kube-api-access-tk429\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.380486 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c8c142-262a-4f8e-aac2-fa61beaee390-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.380534 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77c8c142-262a-4f8e-aac2-fa61beaee390-serving-cert\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.380578 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77c8c142-262a-4f8e-aac2-fa61beaee390-tmp\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.380642 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/77c8c142-262a-4f8e-aac2-fa61beaee390-snapshots\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.380702 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.380656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c8c142-262a-4f8e-aac2-fa61beaee390-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.382100 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.381315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-config\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.382100 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.381645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c8c142-262a-4f8e-aac2-fa61beaee390-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.382100 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.381668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77c8c142-262a-4f8e-aac2-fa61beaee390-tmp\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.382100 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.381897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77c8c142-262a-4f8e-aac2-fa61beaee390-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.382420 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.382144 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/77c8c142-262a-4f8e-aac2-fa61beaee390-snapshots\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.382943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.382914 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-serving-cert\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.386345 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.385108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77c8c142-262a-4f8e-aac2-fa61beaee390-serving-cert\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.390017 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.389990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8f7g\" (UniqueName: \"kubernetes.io/projected/77c8c142-262a-4f8e-aac2-fa61beaee390-kube-api-access-v8f7g\") pod \"insights-operator-76bcd6ccf5-wpxtc\" (UID: \"77c8c142-262a-4f8e-aac2-fa61beaee390\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.390112 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.390040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk429\" (UniqueName: \"kubernetes.io/projected/62f1ef7e-86c7-4222-a2c5-89f2b852fa00-kube-api-access-tk429\") pod \"service-ca-operator-859f94c4b-psrn9\" (UID: \"62f1ef7e-86c7-4222-a2c5-89f2b852fa00\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.443242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.443212 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" Sep 10 17:12:24.450318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.450290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" Sep 10 17:12:24.481497 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.481461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9crp\" (UniqueName: \"kubernetes.io/projected/43627182-feb6-4df9-b293-93645921582f-kube-api-access-r9crp\") pod \"network-check-source-685456655f-pxdwb\" (UID: \"43627182-feb6-4df9-b293-93645921582f\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" Sep 10 17:12:24.491301 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.491277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9crp\" (UniqueName: \"kubernetes.io/projected/43627182-feb6-4df9-b293-93645921582f-kube-api-access-r9crp\") pod \"network-check-source-685456655f-pxdwb\" (UID: \"43627182-feb6-4df9-b293-93645921582f\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" Sep 10 17:12:24.534761 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.534532 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" Sep 10 17:12:24.619757 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.619723 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj"] Sep 10 17:12:24.624405 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:24.623680 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa8506b_482d_456a_bab2_a7c26b3d8e64.slice/crio-edf140dd72c5432a1d78f14c8fe22fe37df8b4d1024fb923ee3abad1acf511b5 WatchSource:0}: Error finding container edf140dd72c5432a1d78f14c8fe22fe37df8b4d1024fb923ee3abad1acf511b5: Status 404 returned error can't find the container with id edf140dd72c5432a1d78f14c8fe22fe37df8b4d1024fb923ee3abad1acf511b5 Sep 10 17:12:24.657048 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.656976 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-tv8w9"] Sep 10 17:12:24.659997 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:24.659971 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a87714_cd43_4cd4_b7a6_d23ad894d1df.slice/crio-a9f7cb314cd470640827d1250791877693c397b50b5048607984f3eec5229083 WatchSource:0}: Error finding container a9f7cb314cd470640827d1250791877693c397b50b5048607984f3eec5229083: Status 404 returned error can't find the container with id a9f7cb314cd470640827d1250791877693c397b50b5048607984f3eec5229083 Sep 10 17:12:24.682089 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.682067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j"] Sep 10 17:12:24.683412 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.683391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.683512 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.683447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:24.683512 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.683483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:24.683618 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.683554 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:12:24.683618 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.683567 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:24.683618 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.683610 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:25.683596556 +0000 UTC m=+51.664676551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : configmap references non-existent config key: service-ca.crt Sep 10 17:12:24.683763 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.683626 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls podName:170c6e7a-0aa9-41f8-a475-904a458b67a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:25.683620105 +0000 UTC m=+51.664700081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6zv7s" (UID: "170c6e7a-0aa9-41f8-a475-904a458b67a0") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:24.683763 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.683638 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:25.683630884 +0000 UTC m=+51.664710860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : secret "router-metrics-certs-default" not found Sep 10 17:12:24.684102 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:24.684081 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1aa361_e7eb_48f9_82e0_6bb7206e0beb.slice/crio-10dccef6571b345d65812091122503192f80b91585fc8a7484f8f435d54f0a0f WatchSource:0}: Error finding container 10dccef6571b345d65812091122503192f80b91585fc8a7484f8f435d54f0a0f: Status 404 returned error can't find the container with id 10dccef6571b345d65812091122503192f80b91585fc8a7484f8f435d54f0a0f Sep 10 17:12:24.706370 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.706329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" event={"ID":"5fa8506b-482d-456a-bab2-a7c26b3d8e64","Type":"ContainerStarted","Data":"edf140dd72c5432a1d78f14c8fe22fe37df8b4d1024fb923ee3abad1acf511b5"} Sep 10 17:12:24.707481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.707439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" event={"ID":"71a87714-cd43-4cd4-b7a6-d23ad894d1df","Type":"ContainerStarted","Data":"a9f7cb314cd470640827d1250791877693c397b50b5048607984f3eec5229083"} Sep 10 17:12:24.708542 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.708523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" event={"ID":"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb","Type":"ContainerStarted","Data":"10dccef6571b345d65812091122503192f80b91585fc8a7484f8f435d54f0a0f"} Sep 10 17:12:24.750927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.750898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9"] Sep 10 17:12:24.753789 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:24.753761 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f1ef7e_86c7_4222_a2c5_89f2b852fa00.slice/crio-79f5d1ed76435409665ba29f6f4942b9888c9df1adc85e2233c03dce83385288 WatchSource:0}: Error finding container 79f5d1ed76435409665ba29f6f4942b9888c9df1adc85e2233c03dce83385288: Status 404 returned error can't find the container with id 79f5d1ed76435409665ba29f6f4942b9888c9df1adc85e2233c03dce83385288 Sep 10 17:12:24.772987 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.772946 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-wpxtc"] Sep 10 17:12:24.775260 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:24.775238 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c8c142_262a_4f8e_aac2_fa61beaee390.slice/crio-92a19365e8355250f0f10435c886f6059a71b59e061a135bcd38e305ecf44311 WatchSource:0}: Error finding container 92a19365e8355250f0f10435c886f6059a71b59e061a135bcd38e305ecf44311: Status 404 returned error can't find the container with id 92a19365e8355250f0f10435c886f6059a71b59e061a135bcd38e305ecf44311 Sep 10 17:12:24.784831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.784809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:24.784961 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.784948 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 17:12:24.785008 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:24.784997 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls podName:5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:25.784984833 +0000 UTC m=+51.766064808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hz8hz" (UID: "5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9") : secret "samples-operator-tls" not found Sep 10 17:12:24.844562 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:24.844522 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pxdwb"] Sep 10 17:12:24.847248 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:24.847226 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43627182_feb6_4df9_b293_93645921582f.slice/crio-fea314038eb831c5539c71d20ccb087cae4f3c4d42ea5822a4305d66055c62c5 WatchSource:0}: Error finding container fea314038eb831c5539c71d20ccb087cae4f3c4d42ea5822a4305d66055c62c5: Status 404 returned error can't find the container with id fea314038eb831c5539c71d20ccb087cae4f3c4d42ea5822a4305d66055c62c5 Sep 10 17:12:25.693989 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:25.693944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:25.694477 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:25.694045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:25.694477 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:25.694095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:25.694477 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:25.694237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:27.694216756 +0000 UTC m=+53.675296739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : configmap references non-existent config key: service-ca.crt Sep 10 17:12:25.694756 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:25.694634 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:25.694756 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:25.694694 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls podName:170c6e7a-0aa9-41f8-a475-904a458b67a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:27.694677749 +0000 UTC m=+53.675757742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6zv7s" (UID: "170c6e7a-0aa9-41f8-a475-904a458b67a0") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:25.694756 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:25.694634 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:12:25.694756 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:25.694734 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:27.694724686 +0000 UTC m=+53.675804667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : secret "router-metrics-certs-default" not found Sep 10 17:12:25.714174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:25.713972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" event={"ID":"43627182-feb6-4df9-b293-93645921582f","Type":"ContainerStarted","Data":"fea314038eb831c5539c71d20ccb087cae4f3c4d42ea5822a4305d66055c62c5"} Sep 10 17:12:25.716169 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:25.716139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" event={"ID":"77c8c142-262a-4f8e-aac2-fa61beaee390","Type":"ContainerStarted","Data":"92a19365e8355250f0f10435c886f6059a71b59e061a135bcd38e305ecf44311"} Sep 10 17:12:25.718233 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:25.718165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" event={"ID":"62f1ef7e-86c7-4222-a2c5-89f2b852fa00","Type":"ContainerStarted","Data":"79f5d1ed76435409665ba29f6f4942b9888c9df1adc85e2233c03dce83385288"} Sep 10 17:12:25.796225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:25.795556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:25.796225 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:25.795776 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 17:12:25.796225 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:25.795843 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls podName:5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:27.795824705 +0000 UTC m=+53.776904685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hz8hz" (UID: "5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9") : secret "samples-operator-tls" not found Sep 10 17:12:26.722132 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:26.722091 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" event={"ID":"5fa8506b-482d-456a-bab2-a7c26b3d8e64","Type":"ContainerStarted","Data":"1162fdbec2ced2e6f18f212b9b027a605c7e7cd26362a06d6c1bbad969222fd7"} Sep 10 17:12:26.741725 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:26.741527 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l2vqj" podStartSLOduration=2.092886888 podStartE2EDuration="3.741505286s" totalImagesPullingTime="1.648618398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:26.740250194 +0000 UTC m=+52.721330195" watchObservedRunningTime="2026-09-10 17:12:26.741505286 +0000 UTC m=+52.722585285" Sep 10 17:12:26.816988 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:26.816947 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:26.877026 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:26.876980 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:27.714296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:27.714254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:27.714500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:27.714318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:27.714500 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:27.714403 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:27.714500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:27.714419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:27.714500 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:27.714489 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls podName:170c6e7a-0aa9-41f8-a475-904a458b67a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:31.714466012 +0000 UTC m=+57.695546011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6zv7s" (UID: "170c6e7a-0aa9-41f8-a475-904a458b67a0") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:27.714685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:27.714516 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:12:27.714685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:27.714518 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:31.714498838 +0000 UTC m=+57.695578821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : configmap references non-existent config key: service-ca.crt Sep 10 17:12:27.714685 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:27.714576 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:31.714560642 +0000 UTC m=+57.695640625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : secret "router-metrics-certs-default" not found Sep 10 17:12:27.815198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:27.815143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:27.815663 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:27.815340 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 17:12:27.815663 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:27.815422 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls podName:5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:31.815401567 +0000 UTC m=+57.796481557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hz8hz" (UID: "5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9") : secret "samples-operator-tls" not found Sep 10 17:12:29.758324 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:29.756867 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" podStartSLOduration=0.875401617 podStartE2EDuration="5.756844437s" totalImagesPullingTime="4.88144282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:29.75612467 +0000 UTC m=+55.737204668" watchObservedRunningTime="2026-09-10 17:12:29.756844437 +0000 UTC m=+55.737924436" Sep 10 17:12:29.959296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:29.959200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:30.024047 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.023959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:30.749226 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.749194 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/0.log" Sep 10 17:12:30.749446 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.749245 2576 generic.go:356] "Generic (PLEG): container finished" podID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" containerID="daef030ddb4e1ef74846cf3ee6ab539e5e7b25968dddd43e797aeedd60fcb642" exitCode=255 Sep 10 17:12:30.749446 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.749331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" event={"ID":"71a87714-cd43-4cd4-b7a6-d23ad894d1df","Type":"ContainerDied","Data":"daef030ddb4e1ef74846cf3ee6ab539e5e7b25968dddd43e797aeedd60fcb642"} Sep 10 17:12:30.749638 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.749599 2576 scope.go:117] "RemoveContainer" containerID="daef030ddb4e1ef74846cf3ee6ab539e5e7b25968dddd43e797aeedd60fcb642" Sep 10 17:12:30.751633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.751610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" event={"ID":"43627182-feb6-4df9-b293-93645921582f","Type":"ContainerStarted","Data":"c7034f7bbebb71a641ab57ce49d42d7cfad605b491df2132e52002c2d6dfb8ea"} Sep 10 17:12:30.754427 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.753973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" event={"ID":"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb","Type":"ContainerStarted","Data":"d4284e2ec4756cc28502879c828a37ff28adc9348fafb28bff0e1b60fb4b7cc7"} Sep 10 17:12:30.755806 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.755778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:30.755924 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.755854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" event={"ID":"77c8c142-262a-4f8e-aac2-fa61beaee390","Type":"ContainerStarted","Data":"574ea931108f4cf65b87e6a54dd57bf1f1fcccad74671b30d6ec744b2160aa8d"} Sep 10 17:12:30.757653 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.757621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" event={"ID":"62f1ef7e-86c7-4222-a2c5-89f2b852fa00","Type":"ContainerStarted","Data":"bec842a629f7aa15854312e750b619f5923e7c4a9a4f2c608776131ab2c5c188"} Sep 10 17:12:30.785122 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.785074 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" podStartSLOduration=2.855096573 podStartE2EDuration="7.78505655s" totalImagesPullingTime="4.929959977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:30.784649083 +0000 UTC m=+56.765729082" watchObservedRunningTime="2026-09-10 17:12:30.78505655 +0000 UTC m=+56.766136526" Sep 10 17:12:30.851700 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.851637 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" podStartSLOduration=2.007912121 podStartE2EDuration="6.851614611s" totalImagesPullingTime="4.84370249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:30.81252101 +0000 UTC m=+56.793601009" watchObservedRunningTime="2026-09-10 17:12:30.851614611 +0000 UTC m=+56.832694610" Sep 10 17:12:30.852079 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.852042 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-pxdwb" podStartSLOduration=2.053134285 podStartE2EDuration="6.852027767s" totalImagesPullingTime="4.798893482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:30.850282339 +0000 UTC m=+56.831362342" watchObservedRunningTime="2026-09-10 17:12:30.852027767 +0000 UTC m=+56.833107766" Sep 10 17:12:30.881161 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.881105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:30.971252 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:30.971160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:31.068258 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.068223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:31.513638 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.513600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:31.617734 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.617689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:31.752952 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.752901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:31.753147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.752975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:31.753147 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.753043 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:31.753147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.753084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:31.753268 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.753158 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls podName:170c6e7a-0aa9-41f8-a475-904a458b67a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.753135426 +0000 UTC m=+65.734215424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6zv7s" (UID: "170c6e7a-0aa9-41f8-a475-904a458b67a0") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:31.753268 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.753162 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:12:31.753268 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.753184 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.753172945 +0000 UTC m=+65.734252922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : configmap references non-existent config key: service-ca.crt Sep 10 17:12:31.753268 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.753204 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.753193085 +0000 UTC m=+65.734273066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : secret "router-metrics-certs-default" not found Sep 10 17:12:31.763363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.763338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/1.log" Sep 10 17:12:31.763811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.763770 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/0.log" Sep 10 17:12:31.763811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.763805 2576 generic.go:356] "Generic (PLEG): container finished" podID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" containerID="13cb059096ccd1d6580ef0b5c4dec18bd25f60c471c032ba13a2e41b12243c2a" exitCode=255 Sep 10 17:12:31.763932 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.763906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" event={"ID":"71a87714-cd43-4cd4-b7a6-d23ad894d1df","Type":"ContainerDied","Data":"13cb059096ccd1d6580ef0b5c4dec18bd25f60c471c032ba13a2e41b12243c2a"} Sep 10 17:12:31.764009 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.763996 2576 scope.go:117] "RemoveContainer" containerID="daef030ddb4e1ef74846cf3ee6ab539e5e7b25968dddd43e797aeedd60fcb642" Sep 10 17:12:31.764398 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.764378 2576 scope.go:117] "RemoveContainer" containerID="13cb059096ccd1d6580ef0b5c4dec18bd25f60c471c032ba13a2e41b12243c2a" Sep 10 17:12:31.764551 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.764529 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-tv8w9_openshift-console-operator(71a87714-cd43-4cd4-b7a6-d23ad894d1df)\"" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" podUID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" Sep 10 17:12:31.854601 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.854220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:31.854601 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.854326 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 17:12:31.854601 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:31.854385 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls podName:5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.854365775 +0000 UTC m=+65.835445756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hz8hz" (UID: "5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9") : secret "samples-operator-tls" not found Sep 10 17:12:31.920745 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.920712 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lp2hc"] Sep 10 17:12:31.937698 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.937659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:31.953589 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.953557 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lp2hc"] Sep 10 17:12:31.953748 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.953664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:31.956047 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.956017 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 17:12:31.956047 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.956034 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2k9xc\"" Sep 10 17:12:31.956198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.956103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 17:12:31.994226 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:31.994195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:32.056583 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.056500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.056583 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.056560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19b86683-66b6-4717-bae8-95e9bed10496-crio-socket\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.056801 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.056772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19b86683-66b6-4717-bae8-95e9bed10496-data-volume\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.056901 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.056855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzqw\" (UniqueName: \"kubernetes.io/projected/19b86683-66b6-4717-bae8-95e9bed10496-kube-api-access-dtzqw\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.057013 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.056924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158155 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.158114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19b86683-66b6-4717-bae8-95e9bed10496-data-volume\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.158172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzqw\" (UniqueName: \"kubernetes.io/projected/19b86683-66b6-4717-bae8-95e9bed10496-kube-api-access-dtzqw\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.158199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.158217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.158247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19b86683-66b6-4717-bae8-95e9bed10496-crio-socket\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.158332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19b86683-66b6-4717-bae8-95e9bed10496-crio-socket\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158569 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.158364 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:12:32.158569 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.158427 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:32.658413203 +0000 UTC m=+58.639493178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : configmap "kube-rbac-proxy" not found Sep 10 17:12:32.158569 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.158369 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:12:32.158569 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.158460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19b86683-66b6-4717-bae8-95e9bed10496-data-volume\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.158569 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.158516 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:32.658497367 +0000 UTC m=+58.639577377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : secret "insights-runtime-extractor-tls" not found Sep 10 17:12:32.167401 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.167367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzqw\" (UniqueName: \"kubernetes.io/projected/19b86683-66b6-4717-bae8-95e9bed10496-kube-api-access-dtzqw\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.662866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.662829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.662866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.662869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:32.663106 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.662972 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:12:32.663106 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.663002 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:12:32.663106 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.663023 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:33.663010541 +0000 UTC m=+59.644090518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : configmap "kube-rbac-proxy" not found Sep 10 17:12:32.663106 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.663077 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:33.663057356 +0000 UTC m=+59.644137345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : secret "insights-runtime-extractor-tls" not found Sep 10 17:12:32.768561 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.768533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/1.log" Sep 10 17:12:32.768863 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:32.768848 2576 scope.go:117] "RemoveContainer" containerID="13cb059096ccd1d6580ef0b5c4dec18bd25f60c471c032ba13a2e41b12243c2a" Sep 10 17:12:32.769043 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:32.769026 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-tv8w9_openshift-console-operator(71a87714-cd43-4cd4-b7a6-d23ad894d1df)\"" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" podUID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" Sep 10 17:12:33.138319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:33.138284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:33.196596 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:33.196563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:33.670671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:33.670637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:33.671663 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:33.671644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:33.671710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:33.671675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:33.671775 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:33.671759 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:12:33.671814 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:33.671792 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:12:33.671814 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:33.671809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:35.671796714 +0000 UTC m=+61.652876691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : configmap "kube-rbac-proxy" not found Sep 10 17:12:33.671903 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:33.671845 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:35.671832288 +0000 UTC m=+61.652912265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : secret "insights-runtime-extractor-tls" not found Sep 10 17:12:33.767684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:33.767648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:34.083929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.083895 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:34.184480 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.184448 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:34.282772 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.282735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:34.359769 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.359702 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:34.359906 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.359784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:34.360139 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.360127 2576 scope.go:117] "RemoveContainer" containerID="13cb059096ccd1d6580ef0b5c4dec18bd25f60c471c032ba13a2e41b12243c2a" Sep 10 17:12:34.360301 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:34.360283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-tv8w9_openshift-console-operator(71a87714-cd43-4cd4-b7a6-d23ad894d1df)\"" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" podUID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" Sep 10 17:12:34.502481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.502447 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d9247_40439c75-55c7-42f5-9d84-6b380daf2e05/dns-node-resolver/0.log" Sep 10 17:12:34.657444 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:34.657365 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvskl" Sep 10 17:12:35.507181 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:35.507151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:35.580708 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:35.580672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:35.689062 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:35.689028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:35.689062 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:35.689066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:35.689322 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:35.689197 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:12:35.689322 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:35.689233 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:12:35.689322 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:35.689267 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.689251191 +0000 UTC m=+65.670331168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : secret "insights-runtime-extractor-tls" not found Sep 10 17:12:35.689322 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:35.689292 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.689274719 +0000 UTC m=+65.670354717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : configmap "kube-rbac-proxy" not found Sep 10 17:12:35.696595 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:35.696572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d2rkj_f6feefb4-04cc-497b-870f-609aab9359f9/node-ca/0.log" Sep 10 17:12:36.498195 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:36.498162 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cndb6_7428d8c1-26fb-4a73-b4c2-ff5cee1533cc/serve-healthcheck-canary/0.log" Sep 10 17:12:37.306578 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:37.306551 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fxk8j_1a1aa361-e7eb-48f9-82e0-6bb7206e0beb/kube-storage-version-migrator-operator/0.log" Sep 10 17:12:37.524677 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:37.524648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:38.386100 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:38.386067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:38.483553 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:38.483507 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:38.553116 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:38.553081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:38.640119 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:38.640042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:38.741227 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:38.741190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:38.841442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:38.841404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:39.028622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.028543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:39.726272 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.726230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:39.726272 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.726273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:39.726772 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.726395 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:12:39.726772 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.726408 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:12:39.726772 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.726461 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:47.72644526 +0000 UTC m=+73.707525240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : secret "insights-runtime-extractor-tls" not found Sep 10 17:12:39.726772 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.726487 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:47.726471298 +0000 UTC m=+73.707551274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : configmap "kube-rbac-proxy" not found Sep 10 17:12:39.826171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.826129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:39.827205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.827185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:39.827270 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.827240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:39.827335 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.827321 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:12:39.827375 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.827366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:39.827417 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.827320 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:39.827454 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.827428 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:55.827408899 +0000 UTC m=+81.808488878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : secret "router-metrics-certs-default" not found Sep 10 17:12:39.827502 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.827467 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls podName:170c6e7a-0aa9-41f8-a475-904a458b67a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:55.827452296 +0000 UTC m=+81.808532286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6zv7s" (UID: "170c6e7a-0aa9-41f8-a475-904a458b67a0") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:12:39.827502 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.827480 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle podName:99a001fe-c41e-457b-b8ff-64fa4da2af08 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:55.827474632 +0000 UTC m=+81.808554607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle") pod "router-default-68d5b75dfd-l6m8l" (UID: "99a001fe-c41e-457b-b8ff-64fa4da2af08") : configmap references non-existent config key: service-ca.crt Sep 10 17:12:39.928797 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:39.928766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:39.928947 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.928927 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 17:12:39.929002 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:39.928993 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls podName:5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:55.928978012 +0000 UTC m=+81.910057989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hz8hz" (UID: "5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9") : secret "samples-operator-tls" not found Sep 10 17:12:40.030807 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.030728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:40.231228 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.231189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:40.231377 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.231261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:40.233605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.233585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmc4z\" (UniqueName: \"kubernetes.io/projected/66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d-kube-api-access-pmc4z\") pod \"network-check-target-t5lgb\" (UID: \"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d\") " pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:40.234211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.234193 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 17:12:40.244272 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.244250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f7c913b-fedc-4ca8-9d95-fa73663da5d4-original-pull-secret\") pod \"global-pull-secret-syncer-8r6bk\" (UID: \"0f7c913b-fedc-4ca8-9d95-fa73663da5d4\") " pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:40.292460 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.292380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6bk" Sep 10 17:12:40.302839 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.302804 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2g8g\"" Sep 10 17:12:40.311026 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.311004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:40.332250 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.332224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:12:40.332390 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.332311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:12:40.332390 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:40.332377 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:12:40.332390 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.332381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:12:40.332527 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:40.332445 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls podName:a9cc5e36-0e94-4097-b240-bd5660695b52 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:12.332427232 +0000 UTC m=+98.313507230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls") pod "dns-default-xbd5r" (UID: "a9cc5e36-0e94-4097-b240-bd5660695b52") : secret "dns-default-metrics-tls" not found Sep 10 17:12:40.332527 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:40.332513 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 17:12:40.332632 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:40.332540 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-849f8f8ffd-n4xtz: secret "image-registry-tls" not found Sep 10 17:12:40.332632 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:40.332619 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls podName:3fd67450-0bb9-4894-8e12-02053f31faf1 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:12.33260215 +0000 UTC m=+98.313682125 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls") pod "image-registry-849f8f8ffd-n4xtz" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1") : secret "image-registry-tls" not found Sep 10 17:12:40.336091 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.336065 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 17:12:40.343024 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:40.343004 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 17:12:40.343107 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:40.343068 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs podName:4721421f-ab96-45e5-8503-ccc0a9a3f83f nodeName:}" failed. No retries permitted until 2026-09-10 17:13:44.343050989 +0000 UTC m=+130.324130965 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs") pod "network-metrics-daemon-555sb" (UID: "4721421f-ab96-45e5-8503-ccc0a9a3f83f") : secret "metrics-daemon-secret" not found Sep 10 17:12:40.452262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.452222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:40.561704 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.561681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8r6bk"] Sep 10 17:12:40.564079 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:40.564052 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7c913b_fedc_4ca8_9d95_fa73663da5d4.slice/crio-38b6676b44f0b69d2ea6ee1d5021de3f28a3897f60a3e738ce338cdcbe9f50ea WatchSource:0}: Error finding container 38b6676b44f0b69d2ea6ee1d5021de3f28a3897f60a3e738ce338cdcbe9f50ea: Status 404 returned error can't find the container with id 38b6676b44f0b69d2ea6ee1d5021de3f28a3897f60a3e738ce338cdcbe9f50ea Sep 10 17:12:40.587117 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.587091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t5lgb"] Sep 10 17:12:40.590957 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:40.590927 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f59ae8_c3b8_4ce6_83b2_cb0dcd4b3b5d.slice/crio-0fbc0910f11032305dfd90542dc90034bb1a370d03d74bedba6e637639bfd1ba WatchSource:0}: Error finding container 0fbc0910f11032305dfd90542dc90034bb1a370d03d74bedba6e637639bfd1ba: Status 404 returned error can't find the container with id 0fbc0910f11032305dfd90542dc90034bb1a370d03d74bedba6e637639bfd1ba Sep 10 17:12:40.793306 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.793269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8r6bk" event={"ID":"0f7c913b-fedc-4ca8-9d95-fa73663da5d4","Type":"ContainerStarted","Data":"38b6676b44f0b69d2ea6ee1d5021de3f28a3897f60a3e738ce338cdcbe9f50ea"} Sep 10 17:12:40.794736 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.794710 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t5lgb" event={"ID":"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d","Type":"ContainerStarted","Data":"93d0a1ac0ee307dbdced65bb300348836aa1e28a4e817c6c5af73b8f476ee997"} Sep 10 17:12:40.794803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.794759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t5lgb" event={"ID":"66f59ae8-c3b8-4ce6-83b2-cb0dcd4b3b5d","Type":"ContainerStarted","Data":"0fbc0910f11032305dfd90542dc90034bb1a370d03d74bedba6e637639bfd1ba"} Sep 10 17:12:40.794852 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.794832 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:12:40.815096 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:40.813485 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t5lgb" podStartSLOduration=65.813462864 podStartE2EDuration="1m5.813462864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:40.811190181 +0000 UTC m=+66.792270179" watchObservedRunningTime="2026-09-10 17:12:40.813462864 +0000 UTC m=+66.794542862" Sep 10 17:12:41.227296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:41.227247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:41.428765 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:41.428731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:41.825128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:41.825077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:42.023058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:42.023017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:42.089827 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:42.089733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:42.399145 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:42.399023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:42.627009 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:42.626975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:42.828263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:42.828218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:42.887388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:42.887352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:43.219915 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:43.219761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:43.822986 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:43.822949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:43.990715 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:43.990665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:44.438688 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:44.438650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:44.507420 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:44.507341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:44.577902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:44.577847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:44.681157 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:44.681112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:44.827819 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:44.827758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:45.225841 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:45.225757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:45.817079 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:45.817035 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8r6bk" event={"ID":"0f7c913b-fedc-4ca8-9d95-fa73663da5d4","Type":"ContainerStarted","Data":"2c914ba15293cd02db29c2681509cf50c44c1bcb5faa57c21c63551b7dad98a7"} Sep 10 17:12:45.839431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:45.839385 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8r6bk" podStartSLOduration=65.871754321 podStartE2EDuration="1m10.839372447s" totalImagesPullingTime="4.967618126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:45.839356882 +0000 UTC m=+71.820436879" watchObservedRunningTime="2026-09-10 17:12:45.839372447 +0000 UTC m=+71.820452445" Sep 10 17:12:46.042816 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:46.042776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:46.233478 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:46.233394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:46.638384 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:46.638349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:46.835341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:46.835297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:47.221740 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.221705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:47.480466 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.480391 2576 scope.go:117] "RemoveContainer" containerID="13cb059096ccd1d6580ef0b5c4dec18bd25f60c471c032ba13a2e41b12243c2a" Sep 10 17:12:47.802302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.802217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:47.802302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.802253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:47.802498 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:47.802366 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:12:47.802498 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:47.802429 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls podName:19b86683-66b6-4717-bae8-95e9bed10496 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:03.80241308 +0000 UTC m=+89.783493059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lp2hc" (UID: "19b86683-66b6-4717-bae8-95e9bed10496") : secret "insights-runtime-extractor-tls" not found Sep 10 17:12:47.802758 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.802741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19b86683-66b6-4717-bae8-95e9bed10496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:12:47.825027 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.825000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:12:47.825442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.825428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/1.log" Sep 10 17:12:47.825503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.825457 2576 generic.go:356] "Generic (PLEG): container finished" podID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" containerID="2a87bae4235f80f82fa3019a52dccd0bc369d8d66136d39a39d082ae51944a0a" exitCode=255 Sep 10 17:12:47.825503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.825483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" event={"ID":"71a87714-cd43-4cd4-b7a6-d23ad894d1df","Type":"ContainerDied","Data":"2a87bae4235f80f82fa3019a52dccd0bc369d8d66136d39a39d082ae51944a0a"} Sep 10 17:12:47.825572 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.825523 2576 scope.go:117] "RemoveContainer" containerID="13cb059096ccd1d6580ef0b5c4dec18bd25f60c471c032ba13a2e41b12243c2a" Sep 10 17:12:47.825846 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.825830 2576 scope.go:117] "RemoveContainer" containerID="2a87bae4235f80f82fa3019a52dccd0bc369d8d66136d39a39d082ae51944a0a" Sep 10 17:12:47.826032 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:47.826015 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-tv8w9_openshift-console-operator(71a87714-cd43-4cd4-b7a6-d23ad894d1df)\"" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" podUID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" Sep 10 17:12:47.831626 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:47.831604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:48.026040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:48.026007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:48.429731 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:48.429696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:48.830396 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:48.830366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:48.831727 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:48.831708 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:12:49.025457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:49.025420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:54.359929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:54.359897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:54.360312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:54.359946 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:12:54.360312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:54.360214 2576 scope.go:117] "RemoveContainer" containerID="2a87bae4235f80f82fa3019a52dccd0bc369d8d66136d39a39d082ae51944a0a" Sep 10 17:12:54.360379 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:12:54.360344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-tv8w9_openshift-console-operator(71a87714-cd43-4cd4-b7a6-d23ad894d1df)\"" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" podUID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" Sep 10 17:12:54.882639 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:54.882602 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:54.971308 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:54.971277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:55.071795 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.071764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:55.174145 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.174052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:55.270934 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.270898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:55.871006 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.870970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:55.871354 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.871027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:55.871354 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.871078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:55.871676 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.871656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a001fe-c41e-457b-b8ff-64fa4da2af08-service-ca-bundle\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:55.873307 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.873285 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/170c6e7a-0aa9-41f8-a475-904a458b67a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6zv7s\" (UID: \"170c6e7a-0aa9-41f8-a475-904a458b67a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:55.873396 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.873294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a001fe-c41e-457b-b8ff-64fa4da2af08-metrics-certs\") pod \"router-default-68d5b75dfd-l6m8l\" (UID: \"99a001fe-c41e-457b-b8ff-64fa4da2af08\") " pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:55.971935 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.971869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:55.974165 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:55.974136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hz8hz\" (UID: \"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:56.033899 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.033850 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bqrg9\"" Sep 10 17:12:56.042099 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.042078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" Sep 10 17:12:56.043326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.043305 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-b9bhx\"" Sep 10 17:12:56.051459 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.051436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:56.145227 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.145170 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9qcbx\"" Sep 10 17:12:56.153314 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.153298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" Sep 10 17:12:56.320062 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.320038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s"] Sep 10 17:12:56.322367 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:56.322339 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170c6e7a_0aa9_41f8_a475_904a458b67a0.slice/crio-75804fff03c83df6af82dbf60cb090ed804ea4b12a87332b07f69aeb986c08e7 WatchSource:0}: Error finding container 75804fff03c83df6af82dbf60cb090ed804ea4b12a87332b07f69aeb986c08e7: Status 404 returned error can't find the container with id 75804fff03c83df6af82dbf60cb090ed804ea4b12a87332b07f69aeb986c08e7 Sep 10 17:12:56.336894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.336861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d5b75dfd-l6m8l"] Sep 10 17:12:56.339150 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:12:56.339127 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a001fe_c41e_457b_b8ff_64fa4da2af08.slice/crio-4dea36fee9b3f0857950693f344d015c6b9be00e37ab44f2f4366321a7d258e3 WatchSource:0}: Error finding container 4dea36fee9b3f0857950693f344d015c6b9be00e37ab44f2f4366321a7d258e3: Status 404 returned error can't find the container with id 4dea36fee9b3f0857950693f344d015c6b9be00e37ab44f2f4366321a7d258e3 Sep 10 17:12:56.420736 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.420613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz"] Sep 10 17:12:56.669815 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.669778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:56.813870 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.813784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:56.855560 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.855529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" event={"ID":"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9","Type":"ContainerStarted","Data":"a8d7b41c21825c7bb1cc43c1ee88b3076dc8ecdf59f82e60960cc180c90542b5"} Sep 10 17:12:56.857152 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.857132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" event={"ID":"99a001fe-c41e-457b-b8ff-64fa4da2af08","Type":"ContainerStarted","Data":"c7a340f4cba83a192c872992bc83a3d98193d0d75c56d48b485173b141b1ccfe"} Sep 10 17:12:56.857261 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.857178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" event={"ID":"99a001fe-c41e-457b-b8ff-64fa4da2af08","Type":"ContainerStarted","Data":"4dea36fee9b3f0857950693f344d015c6b9be00e37ab44f2f4366321a7d258e3"} Sep 10 17:12:56.858334 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.858313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" event={"ID":"170c6e7a-0aa9-41f8-a475-904a458b67a0","Type":"ContainerStarted","Data":"75804fff03c83df6af82dbf60cb090ed804ea4b12a87332b07f69aeb986c08e7"} Sep 10 17:12:56.910897 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:56.910775 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" podStartSLOduration=33.910761298 podStartE2EDuration="33.910761298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:56.909298015 +0000 UTC m=+82.890378013" watchObservedRunningTime="2026-09-10 17:12:56.910761298 +0000 UTC m=+82.891841296" Sep 10 17:12:57.052301 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.052267 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:57.054875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.054856 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:57.087896 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.087857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:57.199834 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.199787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:57.445823 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.445737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:57.543533 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.543473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:57.863194 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.863157 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:57.864499 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:57.864478 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68d5b75dfd-l6m8l" Sep 10 17:12:58.445178 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:58.445142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:58.837342 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:58.837305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:58.899761 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:58.899705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:59.583537 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.583503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:59.685282 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.685203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:59.749372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.749333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:59.875734 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.875693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" event={"ID":"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9","Type":"ContainerStarted","Data":"7c1cb8de03924a6b75bf743c1c8ee9be1ccf9b9bb5b45f5444e71959b79aa50d"} Sep 10 17:12:59.875980 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.875797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" event={"ID":"5e7d92b2-8414-46a0-83d7-3c4c4a57c8b9","Type":"ContainerStarted","Data":"9b5630b1187ffbf97b4a4b8ca02950a89a718a46419aa54d0890740266df2298"} Sep 10 17:12:59.877348 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.877324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" event={"ID":"170c6e7a-0aa9-41f8-a475-904a458b67a0","Type":"ContainerStarted","Data":"e15e3bcb6a7b8ba38e258815857fe9406e4768b13148bc2787c9b77bdd9c5fae"} Sep 10 17:12:59.880633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.880610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:12:59.893067 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.893018 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hz8hz" podStartSLOduration=34.194862576 podStartE2EDuration="36.893002128s" totalImagesPullingTime="2.698139552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:59.892440866 +0000 UTC m=+85.873520865" watchObservedRunningTime="2026-09-10 17:12:59.893002128 +0000 UTC m=+85.874082117" Sep 10 17:12:59.909158 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:12:59.909109 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6zv7s" podStartSLOduration=34.078938527 podStartE2EDuration="36.909090712s" totalImagesPullingTime="2.830152185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:59.908154846 +0000 UTC m=+85.889234845" watchObservedRunningTime="2026-09-10 17:12:59.909090712 +0000 UTC m=+85.890170711" Sep 10 17:13:02.679179 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:02.679142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:02.781689 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:02.781653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:03.709607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:03.709575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:03.809973 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:03.809943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:03.839222 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:03.839171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:13:03.841421 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:03.841393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19b86683-66b6-4717-bae8-95e9bed10496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp2hc\" (UID: \"19b86683-66b6-4717-bae8-95e9bed10496\") " pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:13:04.065213 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.065124 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2k9xc\"" Sep 10 17:13:04.073189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.073166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lp2hc" Sep 10 17:13:04.341871 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.341842 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lp2hc"] Sep 10 17:13:04.344945 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:04.344924 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b86683_66b6_4717_bae8_95e9bed10496.slice/crio-cae65e46124a3f0987c518c124a3604332173ff1463884a956e0a32d25589029 WatchSource:0}: Error finding container cae65e46124a3f0987c518c124a3604332173ff1463884a956e0a32d25589029: Status 404 returned error can't find the container with id cae65e46124a3f0987c518c124a3604332173ff1463884a956e0a32d25589029 Sep 10 17:13:04.733235 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.733141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:04.765468 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.765435 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xccsp"] Sep 10 17:13:04.767904 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.767874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:04.770293 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.770242 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 17:13:04.770465 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.770444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 17:13:04.771381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.771363 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 17:13:04.771600 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.771585 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hfzwm\"" Sep 10 17:13:04.782775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.780678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xccsp"] Sep 10 17:13:04.798082 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.798039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:04.897006 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.896969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp2hc" event={"ID":"19b86683-66b6-4717-bae8-95e9bed10496","Type":"ContainerStarted","Data":"b7f7a7cd16d27f2fe22c20da2929ef70387bc9c79e6fe5905810a5d2f3aa3794"} Sep 10 17:13:04.897168 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.897031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp2hc" event={"ID":"19b86683-66b6-4717-bae8-95e9bed10496","Type":"ContainerStarted","Data":"cae65e46124a3f0987c518c124a3604332173ff1463884a956e0a32d25589029"} Sep 10 17:13:04.922810 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.922776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:04.948283 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.948249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed4e6175-bb3b-4d8b-819a-216728ac1c69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:04.948448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.948319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n568x\" (UniqueName: \"kubernetes.io/projected/ed4e6175-bb3b-4d8b-819a-216728ac1c69-kube-api-access-n568x\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:04.948448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.948343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e6175-bb3b-4d8b-819a-216728ac1c69-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:04.948448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:04.948367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed4e6175-bb3b-4d8b-819a-216728ac1c69-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.049709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.049624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e6175-bb3b-4d8b-819a-216728ac1c69-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.049709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.049668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed4e6175-bb3b-4d8b-819a-216728ac1c69-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.049907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.049722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed4e6175-bb3b-4d8b-819a-216728ac1c69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.049907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.049857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n568x\" (UniqueName: \"kubernetes.io/projected/ed4e6175-bb3b-4d8b-819a-216728ac1c69-kube-api-access-n568x\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.050397 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.050372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed4e6175-bb3b-4d8b-819a-216728ac1c69-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.052176 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.052153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed4e6175-bb3b-4d8b-819a-216728ac1c69-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.052176 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.052168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed4e6175-bb3b-4d8b-819a-216728ac1c69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.058829 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.058810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n568x\" (UniqueName: \"kubernetes.io/projected/ed4e6175-bb3b-4d8b-819a-216728ac1c69-kube-api-access-n568x\") pod \"prometheus-operator-6f4946db56-xccsp\" (UID: \"ed4e6175-bb3b-4d8b-819a-216728ac1c69\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.076786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.076766 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" Sep 10 17:13:05.355446 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.355418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xccsp"] Sep 10 17:13:05.358242 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:05.358215 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4e6175_bb3b_4d8b_819a_216728ac1c69.slice/crio-370101bb21bdc6f76a6c7e654e03362fd63ac2a9ed53b1fb56e8b2dd29b1aafb WatchSource:0}: Error finding container 370101bb21bdc6f76a6c7e654e03362fd63ac2a9ed53b1fb56e8b2dd29b1aafb: Status 404 returned error can't find the container with id 370101bb21bdc6f76a6c7e654e03362fd63ac2a9ed53b1fb56e8b2dd29b1aafb Sep 10 17:13:05.903116 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.903080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp2hc" event={"ID":"19b86683-66b6-4717-bae8-95e9bed10496","Type":"ContainerStarted","Data":"9edfae30c12c95494214c8179853bb19f8e0869cb8c262d3ef65823b5631cf58"} Sep 10 17:13:05.904319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:05.904295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" event={"ID":"ed4e6175-bb3b-4d8b-819a-216728ac1c69","Type":"ContainerStarted","Data":"370101bb21bdc6f76a6c7e654e03362fd63ac2a9ed53b1fb56e8b2dd29b1aafb"} Sep 10 17:13:07.331959 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.331917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:07.480596 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.480566 2576 scope.go:117] "RemoveContainer" containerID="2a87bae4235f80f82fa3019a52dccd0bc369d8d66136d39a39d082ae51944a0a" Sep 10 17:13:07.480755 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:13:07.480738 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-tv8w9_openshift-console-operator(71a87714-cd43-4cd4-b7a6-d23ad894d1df)\"" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" podUID="71a87714-cd43-4cd4-b7a6-d23ad894d1df" Sep 10 17:13:07.680720 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.680640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:07.783604 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.783560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:07.885701 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.885663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:07.914077 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.914043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp2hc" event={"ID":"19b86683-66b6-4717-bae8-95e9bed10496","Type":"ContainerStarted","Data":"3954fa28bee708a450548e4614b55585faa786f391f69b679fec9258bdee50df"} Sep 10 17:13:07.915814 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.915785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" event={"ID":"ed4e6175-bb3b-4d8b-819a-216728ac1c69","Type":"ContainerStarted","Data":"22d4a27d102a4243bd216b13b47f20ddfda5764af74966043b2f9c22cd8826e5"} Sep 10 17:13:07.915944 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.915832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" event={"ID":"ed4e6175-bb3b-4d8b-819a-216728ac1c69","Type":"ContainerStarted","Data":"c6598ee6c9fc8b7ef65c16aa8ffef5ba231d7b5e28a2f5825a3f815b58d90754"} Sep 10 17:13:07.933693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.933613 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lp2hc" podStartSLOduration=34.362494864 podStartE2EDuration="36.933598286s" totalImagesPullingTime="2.571103422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:12:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:07.932473565 +0000 UTC m=+93.913553563" watchObservedRunningTime="2026-09-10 17:13:07.933598286 +0000 UTC m=+93.914678263" Sep 10 17:13:07.954380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.954332 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-xccsp" podStartSLOduration=2.290947192 podStartE2EDuration="3.954317352s" totalImagesPullingTime="1.66337016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:07.953401416 +0000 UTC m=+93.934481414" watchObservedRunningTime="2026-09-10 17:13:07.954317352 +0000 UTC m=+93.935397328" Sep 10 17:13:07.957758 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:07.957733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:09.904549 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:09.904512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.116076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.116032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.203089 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.203002 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5dtf5"] Sep 10 17:13:10.205324 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.205301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.208803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.208784 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-279hs\"" Sep 10 17:13:10.209396 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.209380 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 17:13:10.210185 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.210165 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 17:13:10.211235 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.211217 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 17:13:10.250386 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.250349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.342923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.342873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.393389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/311a0a72-08d7-422d-81c0-aea6b47ea149-metrics-client-ca\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393543 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-textfile\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393543 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393543 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-tls\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-wtmp\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-root\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-sys\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6cd\" (UniqueName: \"kubernetes.io/projected/311a0a72-08d7-422d-81c0-aea6b47ea149-kube-api-access-ww6cd\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.393842 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.393706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.464296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.464210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.494649 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-tls\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494649 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-wtmp\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-root\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-wtmp\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-sys\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-root\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6cd\" (UniqueName: \"kubernetes.io/projected/311a0a72-08d7-422d-81c0-aea6b47ea149-kube-api-access-ww6cd\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/311a0a72-08d7-422d-81c0-aea6b47ea149-sys\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.494979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.494949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.495207 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.495005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/311a0a72-08d7-422d-81c0-aea6b47ea149-metrics-client-ca\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.495207 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.495031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-textfile\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.495281 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.495248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.495532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.495512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-textfile\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.495621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.495516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/311a0a72-08d7-422d-81c0-aea6b47ea149-metrics-client-ca\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.495660 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.495626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.497094 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.497075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-tls\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.497419 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.497400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/311a0a72-08d7-422d-81c0-aea6b47ea149-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.518622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.518591 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6cd\" (UniqueName: \"kubernetes.io/projected/311a0a72-08d7-422d-81c0-aea6b47ea149-kube-api-access-ww6cd\") pod \"node-exporter-5dtf5\" (UID: \"311a0a72-08d7-422d-81c0-aea6b47ea149\") " pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.559369 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.559328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.661279 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.661245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.744371 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.744283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.815035 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.815003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5dtf5" Sep 10 17:13:10.819541 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.819511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:10.826032 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:10.826008 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311a0a72_08d7_422d_81c0_aea6b47ea149.slice/crio-ca10568537e1f65d8b04c9fbbd323834cce7f1d0b9bf997231fc2da750d4a5bf WatchSource:0}: Error finding container ca10568537e1f65d8b04c9fbbd323834cce7f1d0b9bf997231fc2da750d4a5bf: Status 404 returned error can't find the container with id ca10568537e1f65d8b04c9fbbd323834cce7f1d0b9bf997231fc2da750d4a5bf Sep 10 17:13:10.927148 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:10.927103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dtf5" event={"ID":"311a0a72-08d7-422d-81c0-aea6b47ea149","Type":"ContainerStarted","Data":"ca10568537e1f65d8b04c9fbbd323834cce7f1d0b9bf997231fc2da750d4a5bf"} Sep 10 17:13:11.240482 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.240439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:11.338485 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.338444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:11.756779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.756740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:11.802850 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.802812 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t5lgb" Sep 10 17:13:11.857567 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.856826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:11.933114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.933080 2576 generic.go:356] "Generic (PLEG): container finished" podID="311a0a72-08d7-422d-81c0-aea6b47ea149" containerID="4ab3b144ba67218c25dcd908600cde8822a98b40f2ae3afb39170566625f39d3" exitCode=0 Sep 10 17:13:11.933576 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.933162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dtf5" event={"ID":"311a0a72-08d7-422d-81c0-aea6b47ea149","Type":"ContainerDied","Data":"4ab3b144ba67218c25dcd908600cde8822a98b40f2ae3afb39170566625f39d3"} Sep 10 17:13:11.958510 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:11.958480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:12.413556 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.413524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:13:12.413706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.413572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:13:12.415909 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.415863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9cc5e36-0e94-4097-b240-bd5660695b52-metrics-tls\") pod \"dns-default-xbd5r\" (UID: \"a9cc5e36-0e94-4097-b240-bd5660695b52\") " pod="openshift-dns/dns-default-xbd5r" Sep 10 17:13:12.416075 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.416059 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"image-registry-849f8f8ffd-n4xtz\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:13:12.621120 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.621089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ppxr2\"" Sep 10 17:13:12.628822 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.628801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:13:12.645237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.645213 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52r9p\"" Sep 10 17:13:12.653525 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.653503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbd5r" Sep 10 17:13:12.905780 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.905747 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849f8f8ffd-n4xtz"] Sep 10 17:13:12.908908 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:12.908861 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd67450_0bb9_4894_8e12_02053f31faf1.slice/crio-59832f83d3aa865dcb282953e986163eb46688403200c6f8cef21e7b406d6cf5 WatchSource:0}: Error finding container 59832f83d3aa865dcb282953e986163eb46688403200c6f8cef21e7b406d6cf5: Status 404 returned error can't find the container with id 59832f83d3aa865dcb282953e986163eb46688403200c6f8cef21e7b406d6cf5 Sep 10 17:13:12.931711 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.931686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbd5r"] Sep 10 17:13:12.937159 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.937135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" event={"ID":"3fd67450-0bb9-4894-8e12-02053f31faf1","Type":"ContainerStarted","Data":"59832f83d3aa865dcb282953e986163eb46688403200c6f8cef21e7b406d6cf5"} Sep 10 17:13:12.939387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.939112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dtf5" event={"ID":"311a0a72-08d7-422d-81c0-aea6b47ea149","Type":"ContainerStarted","Data":"a971671ef2a925ef16eee72ae6f5d0a86be54e767755fccb8f2bff44effd08ba"} Sep 10 17:13:12.939387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.939181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dtf5" event={"ID":"311a0a72-08d7-422d-81c0-aea6b47ea149","Type":"ContainerStarted","Data":"d4e37181d5f0b838588b4ba73199e93a2b7381512dd5680b2eb16218822e8742"} Sep 10 17:13:12.940084 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:12.940059 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cc5e36_0e94_4097_b240_bd5660695b52.slice/crio-747deaf2ebe3398f44887daecea838386c507e57972fb047b1f86fe7dd084f4f WatchSource:0}: Error finding container 747deaf2ebe3398f44887daecea838386c507e57972fb047b1f86fe7dd084f4f: Status 404 returned error can't find the container with id 747deaf2ebe3398f44887daecea838386c507e57972fb047b1f86fe7dd084f4f Sep 10 17:13:12.958856 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:12.958810 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5dtf5" podStartSLOduration=1.959861501 podStartE2EDuration="2.958795545s" totalImagesPullingTime="929.06815ms" totalInitContainerRuntime="69.865894ms" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:12.957781003 +0000 UTC m=+98.938861014" watchObservedRunningTime="2026-09-10 17:13:12.958795545 +0000 UTC m=+98.939875542" Sep 10 17:13:13.068659 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.068625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:13.196111 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.196018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:13.383527 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.383495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:13.592634 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.592576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:13.945370 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.945292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" event={"ID":"3fd67450-0bb9-4894-8e12-02053f31faf1","Type":"ContainerStarted","Data":"24ba69ff3db353497b0b68aadf894eb8d73312c09a828948c4ff1456ca95e980"} Sep 10 17:13:13.945370 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.945355 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:13:13.946474 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.946454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbd5r" event={"ID":"a9cc5e36-0e94-4097-b240-bd5660695b52","Type":"ContainerStarted","Data":"747deaf2ebe3398f44887daecea838386c507e57972fb047b1f86fe7dd084f4f"} Sep 10 17:13:13.967833 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:13.967785 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" podStartSLOduration=98.967770681 podStartE2EDuration="1m38.967770681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:13.967566463 +0000 UTC m=+99.948646460" watchObservedRunningTime="2026-09-10 17:13:13.967770681 +0000 UTC m=+99.948850658" Sep 10 17:13:14.613605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.613575 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67bcd4c8f7-qn52k"] Sep 10 17:13:14.616371 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.616348 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.621081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.621050 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 17:13:14.622309 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.622286 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 17:13:14.622413 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.622311 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eqa1qgr80j3ob\"" Sep 10 17:13:14.622413 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.622289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cxjll\"" Sep 10 17:13:14.622529 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.622289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 17:13:14.622667 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.622646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 17:13:14.641765 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.641719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67bcd4c8f7-qn52k"] Sep 10 17:13:14.736454 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.736417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-audit-log\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.736605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.736506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-metrics-server-audit-profiles\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.736605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.736543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.736731 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.736587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqbtz\" (UniqueName: \"kubernetes.io/projected/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-kube-api-access-bqbtz\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.736731 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.736654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-secret-metrics-server-client-certs\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.736823 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.736734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-client-ca-bundle\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.736823 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.736755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-secret-metrics-server-tls\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.837991 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.837952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-metrics-server-audit-profiles\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838136 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.837998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838136 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.838041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqbtz\" (UniqueName: \"kubernetes.io/projected/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-kube-api-access-bqbtz\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838212 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.838137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-secret-metrics-server-client-certs\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838212 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.838204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-client-ca-bundle\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838290 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.838233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-secret-metrics-server-tls\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838290 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.838267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-audit-log\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838650 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.838622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-audit-log\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.838822 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.838804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.839108 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.839083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-metrics-server-audit-profiles\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.840651 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.840623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-secret-metrics-server-tls\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.840831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.840809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-client-ca-bundle\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.840914 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.840868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-secret-metrics-server-client-certs\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.849534 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.849477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqbtz\" (UniqueName: \"kubernetes.io/projected/99e3d4da-cacb-4d4d-82ce-2634e4f90cd3-kube-api-access-bqbtz\") pod \"metrics-server-67bcd4c8f7-qn52k\" (UID: \"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3\") " pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.921862 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.921817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:14.928535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.928506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:14.953122 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.953087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbd5r" event={"ID":"a9cc5e36-0e94-4097-b240-bd5660695b52","Type":"ContainerStarted","Data":"fd595449f95a1d74ca9b3bbe3a99afb012ec035a145f95627d4403b3b68726c2"} Sep 10 17:13:14.953491 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.953185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbd5r" event={"ID":"a9cc5e36-0e94-4097-b240-bd5660695b52","Type":"ContainerStarted","Data":"cd83cdbf9fefb7cd367f37428abc62d08a2edf04da2e06cfa11f525ddfac577c"} Sep 10 17:13:14.953491 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.953349 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xbd5r" Sep 10 17:13:14.972056 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:14.971998 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xbd5r" podStartSLOduration=65.315272468 podStartE2EDuration="1m6.97197703s" totalImagesPullingTime="1.656704562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:14.970811866 +0000 UTC m=+100.951891910" watchObservedRunningTime="2026-09-10 17:13:14.97197703 +0000 UTC m=+100.953057029" Sep 10 17:13:15.201333 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:15.201250 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67bcd4c8f7-qn52k"] Sep 10 17:13:15.204969 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:15.204933 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e3d4da_cacb_4d4d_82ce_2634e4f90cd3.slice/crio-c9d21817abadff94e7e4ba4494a43f7bc4cfb5a1ea44af9f25551784038d89c0 WatchSource:0}: Error finding container c9d21817abadff94e7e4ba4494a43f7bc4cfb5a1ea44af9f25551784038d89c0: Status 404 returned error can't find the container with id c9d21817abadff94e7e4ba4494a43f7bc4cfb5a1ea44af9f25551784038d89c0 Sep 10 17:13:15.209096 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:15.209060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:15.958057 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:15.958015 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" event={"ID":"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3","Type":"ContainerStarted","Data":"c9d21817abadff94e7e4ba4494a43f7bc4cfb5a1ea44af9f25551784038d89c0"} Sep 10 17:13:16.419270 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:16.419231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:16.580602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:16.580519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:16.676484 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:16.676399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:16.963579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:16.963491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" event={"ID":"99e3d4da-cacb-4d4d-82ce-2634e4f90cd3","Type":"ContainerStarted","Data":"4f771b0af917f01c282098c59fcfeaf108acebdb50f9d8243bb9b3953831ceee"} Sep 10 17:13:16.990513 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:16.990460 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" podStartSLOduration=1.421931338 podStartE2EDuration="2.990442711s" totalImagesPullingTime="1.568511373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:16.988422982 +0000 UTC m=+102.969502984" watchObservedRunningTime="2026-09-10 17:13:16.990442711 +0000 UTC m=+102.971522709" Sep 10 17:13:17.668635 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:17.668585 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-849f8f8ffd-n4xtz"] Sep 10 17:13:17.773067 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:17.773027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:17.870648 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:17.870610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:17.971192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:17.971099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:18.464431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:18.464395 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:18.544765 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:18.544727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:20.481365 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:20.481335 2576 scope.go:117] "RemoveContainer" containerID="2a87bae4235f80f82fa3019a52dccd0bc369d8d66136d39a39d082ae51944a0a" Sep 10 17:13:20.979374 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:20.979346 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:13:20.979566 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:20.979421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" event={"ID":"71a87714-cd43-4cd4-b7a6-d23ad894d1df","Type":"ContainerStarted","Data":"59f76259204c8b6785df62d6f13d1273590cd26def72b652dce3ee68591c4530"} Sep 10 17:13:20.979692 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:20.979671 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:13:20.997242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:20.997178 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" podStartSLOduration=53.043450646 podStartE2EDuration="57.997164883s" totalImagesPullingTime="4.953714237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:20.995583357 +0000 UTC m=+106.976663369" watchObservedRunningTime="2026-09-10 17:13:20.997164883 +0000 UTC m=+106.978244880" Sep 10 17:13:21.164818 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.164783 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-tv8w9" Sep 10 17:13:21.366374 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.366336 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-6cvjc"] Sep 10 17:13:21.368848 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.368830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6cvjc" Sep 10 17:13:21.372747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.372717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 17:13:21.373187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.373161 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wbrhg\"" Sep 10 17:13:21.373296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.373189 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 17:13:21.382675 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.382654 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6cvjc"] Sep 10 17:13:21.385560 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.385538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m229\" (UniqueName: \"kubernetes.io/projected/01237739-3e8f-4ded-af03-8f4f92ce821d-kube-api-access-6m229\") pod \"downloads-85d98c998f-6cvjc\" (UID: \"01237739-3e8f-4ded-af03-8f4f92ce821d\") " pod="openshift-console/downloads-85d98c998f-6cvjc" Sep 10 17:13:21.387210 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.387186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:21.471767 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.471728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:21.485950 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.485906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m229\" (UniqueName: \"kubernetes.io/projected/01237739-3e8f-4ded-af03-8f4f92ce821d-kube-api-access-6m229\") pod \"downloads-85d98c998f-6cvjc\" (UID: \"01237739-3e8f-4ded-af03-8f4f92ce821d\") " pod="openshift-console/downloads-85d98c998f-6cvjc" Sep 10 17:13:21.494929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.494899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m229\" (UniqueName: \"kubernetes.io/projected/01237739-3e8f-4ded-af03-8f4f92ce821d-kube-api-access-6m229\") pod \"downloads-85d98c998f-6cvjc\" (UID: \"01237739-3e8f-4ded-af03-8f4f92ce821d\") " pod="openshift-console/downloads-85d98c998f-6cvjc" Sep 10 17:13:21.673415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.673332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:21.677490 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.677453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6cvjc" Sep 10 17:13:21.740841 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.739626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:21.855833 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.855789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:21.960399 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.960330 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6cvjc"] Sep 10 17:13:21.963358 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:21.963329 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01237739_3e8f_4ded_af03_8f4f92ce821d.slice/crio-f040642564e6cd8c7a9d9e5ef93d9e1f1cc675c41ebdb65ba092cf65660054a0 WatchSource:0}: Error finding container f040642564e6cd8c7a9d9e5ef93d9e1f1cc675c41ebdb65ba092cf65660054a0: Status 404 returned error can't find the container with id f040642564e6cd8c7a9d9e5ef93d9e1f1cc675c41ebdb65ba092cf65660054a0 Sep 10 17:13:21.984789 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:21.984763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6cvjc" event={"ID":"01237739-3e8f-4ded-af03-8f4f92ce821d","Type":"ContainerStarted","Data":"f040642564e6cd8c7a9d9e5ef93d9e1f1cc675c41ebdb65ba092cf65660054a0"} Sep 10 17:13:22.031472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:22.031439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:22.239594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:22.239489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:22.339629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:22.339584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:22.839107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:22.839077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:23.246184 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:23.246083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:23.344147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:23.344113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:23.445421 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:23.445381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:24.961697 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:24.961662 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xbd5r" Sep 10 17:13:27.173175 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:27.173138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:27.272664 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:27.272628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:27.674808 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:27.674781 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:13:31.816114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.816080 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74dcb98c9b-89gpr"] Sep 10 17:13:31.820150 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.820126 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.823392 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.823358 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 17:13:31.823536 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.823368 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 17:13:31.824470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.824441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 17:13:31.824592 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.824441 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 17:13:31.824592 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.824505 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vgkjp\"" Sep 10 17:13:31.824592 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.824441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 17:13:31.834971 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.834948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dcb98c9b-89gpr"] Sep 10 17:13:31.879159 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.879121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-oauth-serving-cert\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.879347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.879175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-oauth-config\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.879347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.879208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-service-ca\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.879347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.879254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbj2l\" (UniqueName: \"kubernetes.io/projected/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-kube-api-access-wbj2l\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.879347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.879310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-serving-cert\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.879347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.879329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-config\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.980609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.980571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-oauth-config\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.980796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.980630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-service-ca\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.980796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.980667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbj2l\" (UniqueName: \"kubernetes.io/projected/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-kube-api-access-wbj2l\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.980796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.980718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-serving-cert\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.980796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.980743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-config\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.981018 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.980948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-oauth-serving-cert\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.981457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.981407 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-service-ca\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.981575 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.981454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-config\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.981730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.981708 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-oauth-serving-cert\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.983206 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.983185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-serving-cert\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.984060 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.984042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-oauth-config\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:31.994288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:31.994258 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbj2l\" (UniqueName: \"kubernetes.io/projected/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-kube-api-access-wbj2l\") pod \"console-74dcb98c9b-89gpr\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:32.130174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:32.130078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:32.414025 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:32.413997 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dcb98c9b-89gpr"] Sep 10 17:13:32.416765 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:32.416741 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4648ac7e_e79a_4267_87bf_5fd7b87daf7c.slice/crio-ac30191bc586d4084b4559f76d2e0af9c45e1b1534960a886c5fcf687e8d0393 WatchSource:0}: Error finding container ac30191bc586d4084b4559f76d2e0af9c45e1b1534960a886c5fcf687e8d0393: Status 404 returned error can't find the container with id ac30191bc586d4084b4559f76d2e0af9c45e1b1534960a886c5fcf687e8d0393 Sep 10 17:13:33.035162 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:33.035119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dcb98c9b-89gpr" event={"ID":"4648ac7e-e79a-4267-87bf-5fd7b87daf7c","Type":"ContainerStarted","Data":"ac30191bc586d4084b4559f76d2e0af9c45e1b1534960a886c5fcf687e8d0393"} Sep 10 17:13:33.269722 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:33.269672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:33.325820 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:33.325774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:33.432748 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:33.432711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:33.489873 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:33.489832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:34.096517 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:34.096414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:34.155716 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:34.155676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:34.929672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:34.929636 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:34.929970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:34.929728 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:39.622423 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:39.622392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:39.723403 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:39.723367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:39.784044 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:39.783984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:40.071435 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:40.071391 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6cvjc" event={"ID":"01237739-3e8f-4ded-af03-8f4f92ce821d","Type":"ContainerStarted","Data":"3d887ef8413068c78cec78b7338ecc79f53d72fcd66990763e868ad592671fdd"} Sep 10 17:13:40.072258 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:40.072233 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-6cvjc" Sep 10 17:13:40.086590 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:40.086563 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-6cvjc" Sep 10 17:13:40.090287 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:40.090237 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-6cvjc" podStartSLOduration=1.472467829 podStartE2EDuration="19.090221456s" totalImagesPullingTime="17.617753627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:40.089146549 +0000 UTC m=+126.070226549" watchObservedRunningTime="2026-09-10 17:13:40.090221456 +0000 UTC m=+126.071301454" Sep 10 17:13:41.078813 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.078744 2576 generic.go:356] "Generic (PLEG): container finished" podID="77c8c142-262a-4f8e-aac2-fa61beaee390" containerID="574ea931108f4cf65b87e6a54dd57bf1f1fcccad74671b30d6ec744b2160aa8d" exitCode=0 Sep 10 17:13:41.079300 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.078913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" event={"ID":"77c8c142-262a-4f8e-aac2-fa61beaee390","Type":"ContainerDied","Data":"574ea931108f4cf65b87e6a54dd57bf1f1fcccad74671b30d6ec744b2160aa8d"} Sep 10 17:13:41.079422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.079394 2576 scope.go:117] "RemoveContainer" containerID="574ea931108f4cf65b87e6a54dd57bf1f1fcccad74671b30d6ec744b2160aa8d" Sep 10 17:13:41.274624 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.274586 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-774cd57cb9-5rbf7"] Sep 10 17:13:41.280260 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.279863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.294665 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.294269 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 17:13:41.302160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.302091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774cd57cb9-5rbf7"] Sep 10 17:13:41.378388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.377905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-config\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.378388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.377961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-serving-cert\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.378388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.378006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-oauth-serving-cert\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.378388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.378073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-service-ca\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.378388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.378120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2jd\" (UniqueName: \"kubernetes.io/projected/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-kube-api-access-9s2jd\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.378388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.378163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-oauth-config\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.378388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.378183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-trusted-ca-bundle\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.479420 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.479377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-service-ca\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.479674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.479455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2jd\" (UniqueName: \"kubernetes.io/projected/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-kube-api-access-9s2jd\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.479674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.479509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-oauth-config\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.479674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.479534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-trusted-ca-bundle\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.479674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.479611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-config\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.479674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.479636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-serving-cert\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.479674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.479663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-oauth-serving-cert\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.481571 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.480525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-oauth-serving-cert\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.481571 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.481183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-service-ca\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.481764 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.481631 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-trusted-ca-bundle\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.481764 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.481734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-config\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.484746 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.484699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-serving-cert\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.484874 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.484758 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-oauth-config\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.491730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.491672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2jd\" (UniqueName: \"kubernetes.io/projected/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-kube-api-access-9s2jd\") pod \"console-774cd57cb9-5rbf7\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:41.600656 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:41.600362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:42.088312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:42.088271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wpxtc" event={"ID":"77c8c142-262a-4f8e-aac2-fa61beaee390","Type":"ContainerStarted","Data":"f765797c839f20c10df51f9f7c89c90478934e33283c48f5ec4dc2cf6e3ecc51"} Sep 10 17:13:42.687989 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:42.687944 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" podUID="3fd67450-0bb9-4894-8e12-02053f31faf1" containerName="registry" containerID="cri-o://24ba69ff3db353497b0b68aadf894eb8d73312c09a828948c4ff1456ca95e980" gracePeriod=30 Sep 10 17:13:43.051426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.051344 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774cd57cb9-5rbf7"] Sep 10 17:13:43.094302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.094265 2576 generic.go:356] "Generic (PLEG): container finished" podID="3fd67450-0bb9-4894-8e12-02053f31faf1" containerID="24ba69ff3db353497b0b68aadf894eb8d73312c09a828948c4ff1456ca95e980" exitCode=0 Sep 10 17:13:43.094691 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.094348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" event={"ID":"3fd67450-0bb9-4894-8e12-02053f31faf1","Type":"ContainerDied","Data":"24ba69ff3db353497b0b68aadf894eb8d73312c09a828948c4ff1456ca95e980"} Sep 10 17:13:43.152990 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:43.152953 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7f59c3_ba22_41a2_9ef2_c3a20e54c283.slice/crio-79caf2ba114ed8d28f7968b533facec89d102d09c9dea5e71d40675866432787 WatchSource:0}: Error finding container 79caf2ba114ed8d28f7968b533facec89d102d09c9dea5e71d40675866432787: Status 404 returned error can't find the container with id 79caf2ba114ed8d28f7968b533facec89d102d09c9dea5e71d40675866432787 Sep 10 17:13:43.193991 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.193958 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:13:43.296461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296436 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-bound-sa-token\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296569 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296481 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-certificates\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296569 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-image-registry-private-configuration\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296572 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296609 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-trusted-ca\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296669 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mxq2\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-kube-api-access-7mxq2\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296742 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fd67450-0bb9-4894-8e12-02053f31faf1-ca-trust-extracted\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-installation-pull-secrets\") pod \"3fd67450-0bb9-4894-8e12-02053f31faf1\" (UID: \"3fd67450-0bb9-4894-8e12-02053f31faf1\") " Sep 10 17:13:43.296956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.296932 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:13:43.297195 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.297067 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-certificates\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:43.297262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.297238 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:13:43.301597 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.301530 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:13:43.302181 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.302149 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-kube-api-access-7mxq2" (OuterVolumeSpecName: "kube-api-access-7mxq2") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "kube-api-access-7mxq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:13:43.302278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.302226 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:13:43.302506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.302480 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:13:43.304943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.304912 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:13:43.308141 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.308115 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd67450-0bb9-4894-8e12-02053f31faf1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3fd67450-0bb9-4894-8e12-02053f31faf1" (UID: "3fd67450-0bb9-4894-8e12-02053f31faf1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:13:43.398450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.398412 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fd67450-0bb9-4894-8e12-02053f31faf1-ca-trust-extracted\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:43.398450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.398445 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-installation-pull-secrets\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:43.398450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.398457 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-bound-sa-token\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:43.398706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.398466 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fd67450-0bb9-4894-8e12-02053f31faf1-image-registry-private-configuration\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:43.398706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.398474 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-registry-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:43.398706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.398482 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fd67450-0bb9-4894-8e12-02053f31faf1-trusted-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:43.398706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:43.398489 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxq2\" (UniqueName: \"kubernetes.io/projected/3fd67450-0bb9-4894-8e12-02053f31faf1-kube-api-access-7mxq2\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:13:44.101030 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.100996 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" Sep 10 17:13:44.101584 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.100988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849f8f8ffd-n4xtz" event={"ID":"3fd67450-0bb9-4894-8e12-02053f31faf1","Type":"ContainerDied","Data":"59832f83d3aa865dcb282953e986163eb46688403200c6f8cef21e7b406d6cf5"} Sep 10 17:13:44.101584 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.101146 2576 scope.go:117] "RemoveContainer" containerID="24ba69ff3db353497b0b68aadf894eb8d73312c09a828948c4ff1456ca95e980" Sep 10 17:13:44.103400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.103370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dcb98c9b-89gpr" event={"ID":"4648ac7e-e79a-4267-87bf-5fd7b87daf7c","Type":"ContainerStarted","Data":"7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002"} Sep 10 17:13:44.105329 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.105302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774cd57cb9-5rbf7" event={"ID":"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283","Type":"ContainerStarted","Data":"37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17"} Sep 10 17:13:44.105432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.105350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774cd57cb9-5rbf7" event={"ID":"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283","Type":"ContainerStarted","Data":"79caf2ba114ed8d28f7968b533facec89d102d09c9dea5e71d40675866432787"} Sep 10 17:13:44.125294 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.125246 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-774cd57cb9-5rbf7" podStartSLOduration=3.125228136 podStartE2EDuration="3.125228136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:44.124995161 +0000 UTC m=+130.106075160" watchObservedRunningTime="2026-09-10 17:13:44.125228136 +0000 UTC m=+130.106308135" Sep 10 17:13:44.143295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.143236 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74dcb98c9b-89gpr" podStartSLOduration=2.375246229 podStartE2EDuration="13.14321449s" totalImagesPullingTime="10.767968261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:44.14175545 +0000 UTC m=+130.122835449" watchObservedRunningTime="2026-09-10 17:13:44.14321449 +0000 UTC m=+130.124294489" Sep 10 17:13:44.155602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.155559 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-849f8f8ffd-n4xtz"] Sep 10 17:13:44.159294 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.159268 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-849f8f8ffd-n4xtz"] Sep 10 17:13:44.409369 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.409287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:13:44.411970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.411944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4721421f-ab96-45e5-8503-ccc0a9a3f83f-metrics-certs\") pod \"network-metrics-daemon-555sb\" (UID: \"4721421f-ab96-45e5-8503-ccc0a9a3f83f\") " pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:13:44.491199 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.491161 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fd67450-0bb9-4894-8e12-02053f31faf1" path="/var/lib/kubelet/pods/3fd67450-0bb9-4894-8e12-02053f31faf1/volumes" Sep 10 17:13:44.507807 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.507777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vgb7\"" Sep 10 17:13:44.516015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.515986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-555sb" Sep 10 17:13:44.825433 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.825396 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-555sb"] Sep 10 17:13:44.829568 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:13:44.829528 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4721421f_ab96_45e5_8503_ccc0a9a3f83f.slice/crio-2d600a907671ca5683866999ba7fe43862b285729e956050174a5c0b807a9ae8 WatchSource:0}: Error finding container 2d600a907671ca5683866999ba7fe43862b285729e956050174a5c0b807a9ae8: Status 404 returned error can't find the container with id 2d600a907671ca5683866999ba7fe43862b285729e956050174a5c0b807a9ae8 Sep 10 17:13:44.869577 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.869535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:44.926390 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:44.926353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:45.112412 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:45.112319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-555sb" event={"ID":"4721421f-ab96-45e5-8503-ccc0a9a3f83f","Type":"ContainerStarted","Data":"2d600a907671ca5683866999ba7fe43862b285729e956050174a5c0b807a9ae8"} Sep 10 17:13:47.124123 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:47.124030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-555sb" event={"ID":"4721421f-ab96-45e5-8503-ccc0a9a3f83f","Type":"ContainerStarted","Data":"f50c055ef96fedaa8dc2badf33e5eadc20ae9ec16db2bcea7a5cd9f62b2be828"} Sep 10 17:13:47.124123 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:47.124091 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-555sb" event={"ID":"4721421f-ab96-45e5-8503-ccc0a9a3f83f","Type":"ContainerStarted","Data":"aefd3e4063889ab9443cc6d3795e19d5b443a3853cb5bc4726919c679d6b6bd1"} Sep 10 17:13:47.143751 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:47.143690 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-555sb" podStartSLOduration=130.41951473 podStartE2EDuration="2m12.143672265s" totalImagesPullingTime="1.724157535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:47.141894261 +0000 UTC m=+133.122974254" watchObservedRunningTime="2026-09-10 17:13:47.143672265 +0000 UTC m=+133.124752242" Sep 10 17:13:47.245648 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:47.245610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:47.324939 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:47.324898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:50.018136 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:50.018101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:50.118673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:50.118634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:51.149775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:51.149743 2576 generic.go:356] "Generic (PLEG): container finished" podID="62f1ef7e-86c7-4222-a2c5-89f2b852fa00" containerID="bec842a629f7aa15854312e750b619f5923e7c4a9a4f2c608776131ab2c5c188" exitCode=0 Sep 10 17:13:51.150164 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:51.149819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" event={"ID":"62f1ef7e-86c7-4222-a2c5-89f2b852fa00","Type":"ContainerDied","Data":"bec842a629f7aa15854312e750b619f5923e7c4a9a4f2c608776131ab2c5c188"} Sep 10 17:13:51.150211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:51.150176 2576 scope.go:117] "RemoveContainer" containerID="bec842a629f7aa15854312e750b619f5923e7c4a9a4f2c608776131ab2c5c188" Sep 10 17:13:51.601064 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:51.601025 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:51.601233 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:51.601100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:51.606062 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:51.606039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:52.130998 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.130964 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:52.131235 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.131022 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:52.135748 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.135726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:52.156445 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.156375 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psrn9" event={"ID":"62f1ef7e-86c7-4222-a2c5-89f2b852fa00","Type":"ContainerStarted","Data":"c48c9afd054c38d386cc606607164c47e1213ee3fafb575efc529f47d99bd84f"} Sep 10 17:13:52.161556 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.161531 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:13:52.161997 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.161974 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:13:52.239338 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.239307 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dcb98c9b-89gpr"] Sep 10 17:13:52.328758 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.328720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:52.433007 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.432917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:52.498341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:52.498314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:53.272745 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:53.272704 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:53.334770 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:53.334740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:13:54.934894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:54.934850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:54.938599 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:54.938575 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67bcd4c8f7-qn52k" Sep 10 17:13:56.173681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:56.173644 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a1aa361-e7eb-48f9-82e0-6bb7206e0beb" containerID="d4284e2ec4756cc28502879c828a37ff28adc9348fafb28bff0e1b60fb4b7cc7" exitCode=0 Sep 10 17:13:56.174089 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:56.173720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" event={"ID":"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb","Type":"ContainerDied","Data":"d4284e2ec4756cc28502879c828a37ff28adc9348fafb28bff0e1b60fb4b7cc7"} Sep 10 17:13:56.174154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:56.174140 2576 scope.go:117] "RemoveContainer" containerID="d4284e2ec4756cc28502879c828a37ff28adc9348fafb28bff0e1b60fb4b7cc7" Sep 10 17:13:57.179726 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:13:57.179695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fxk8j" event={"ID":"1a1aa361-e7eb-48f9-82e0-6bb7206e0beb","Type":"ContainerStarted","Data":"602ca1dfb38dfa07ad9c2fbc150ce79ab9f4a4c11a95b7470a90d1ffbe31c291"} Sep 10 17:14:03.432845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:03.432804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:03.486917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:03.486871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:18.435338 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:18.435299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:18.492129 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:18.492096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:19.184117 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.184071 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74dcb98c9b-89gpr" podUID="4648ac7e-e79a-4267-87bf-5fd7b87daf7c" containerName="console" containerID="cri-o://7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002" gracePeriod=15 Sep 10 17:14:19.459952 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.459859 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:14:19.489963 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.489935 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-oauth-config\") pod \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " Sep 10 17:14:19.489963 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.489965 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-serving-cert\") pod \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " Sep 10 17:14:19.490158 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.489988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-service-ca\") pod \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " Sep 10 17:14:19.490158 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.490019 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-config\") pod \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " Sep 10 17:14:19.490158 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.490142 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbj2l\" (UniqueName: \"kubernetes.io/projected/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-kube-api-access-wbj2l\") pod \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " Sep 10 17:14:19.490308 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.490178 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-oauth-serving-cert\") pod \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\" (UID: \"4648ac7e-e79a-4267-87bf-5fd7b87daf7c\") " Sep 10 17:14:19.490473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.490403 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-service-ca" (OuterVolumeSpecName: "service-ca") pod "4648ac7e-e79a-4267-87bf-5fd7b87daf7c" (UID: "4648ac7e-e79a-4267-87bf-5fd7b87daf7c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:14:19.490473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.490412 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-config" (OuterVolumeSpecName: "console-config") pod "4648ac7e-e79a-4267-87bf-5fd7b87daf7c" (UID: "4648ac7e-e79a-4267-87bf-5fd7b87daf7c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:14:19.490796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.490672 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4648ac7e-e79a-4267-87bf-5fd7b87daf7c" (UID: "4648ac7e-e79a-4267-87bf-5fd7b87daf7c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:14:19.492437 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.492411 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-kube-api-access-wbj2l" (OuterVolumeSpecName: "kube-api-access-wbj2l") pod "4648ac7e-e79a-4267-87bf-5fd7b87daf7c" (UID: "4648ac7e-e79a-4267-87bf-5fd7b87daf7c"). InnerVolumeSpecName "kube-api-access-wbj2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:14:19.492526 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.492432 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4648ac7e-e79a-4267-87bf-5fd7b87daf7c" (UID: "4648ac7e-e79a-4267-87bf-5fd7b87daf7c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:14:19.492526 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.492463 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4648ac7e-e79a-4267-87bf-5fd7b87daf7c" (UID: "4648ac7e-e79a-4267-87bf-5fd7b87daf7c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:14:19.591054 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.591016 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:14:19.591054 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.591046 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbj2l\" (UniqueName: \"kubernetes.io/projected/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-kube-api-access-wbj2l\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:14:19.591054 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.591057 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-oauth-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:14:19.591054 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.591066 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-oauth-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:14:19.591316 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.591074 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-console-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:14:19.591316 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:19.591083 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4648ac7e-e79a-4267-87bf-5fd7b87daf7c-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:14:20.268111 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.268075 2576 generic.go:356] "Generic (PLEG): container finished" podID="4648ac7e-e79a-4267-87bf-5fd7b87daf7c" containerID="7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002" exitCode=0 Sep 10 17:14:20.268271 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.268125 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dcb98c9b-89gpr" Sep 10 17:14:20.268271 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.268140 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dcb98c9b-89gpr" event={"ID":"4648ac7e-e79a-4267-87bf-5fd7b87daf7c","Type":"ContainerDied","Data":"7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002"} Sep 10 17:14:20.268271 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.268192 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dcb98c9b-89gpr" event={"ID":"4648ac7e-e79a-4267-87bf-5fd7b87daf7c","Type":"ContainerDied","Data":"ac30191bc586d4084b4559f76d2e0af9c45e1b1534960a886c5fcf687e8d0393"} Sep 10 17:14:20.268271 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.268219 2576 scope.go:117] "RemoveContainer" containerID="7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002" Sep 10 17:14:20.276607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.276590 2576 scope.go:117] "RemoveContainer" containerID="7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002" Sep 10 17:14:20.276887 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:14:20.276859 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002\": container with ID starting with 7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002 not found: ID does not exist" containerID="7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002" Sep 10 17:14:20.276930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.276903 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002"} err="failed to get container status \"7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002\": rpc error: code = NotFound desc = could not find container \"7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002\": container with ID starting with 7efd4f1d664b3547b291bad2389cada41c31f90ab9f1a80e9761167885ed2002 not found: ID does not exist" Sep 10 17:14:20.288570 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.288549 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dcb98c9b-89gpr"] Sep 10 17:14:20.292248 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.292228 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74dcb98c9b-89gpr"] Sep 10 17:14:20.485960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:20.485928 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4648ac7e-e79a-4267-87bf-5fd7b87daf7c" path="/var/lib/kubelet/pods/4648ac7e-e79a-4267-87bf-5fd7b87daf7c/volumes" Sep 10 17:14:29.445172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:29.445083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:29.748257 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:29.748155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:38.224202 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.224170 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b4f469b7-r2f5n"] Sep 10 17:14:38.224621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.224478 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4648ac7e-e79a-4267-87bf-5fd7b87daf7c" containerName="console" Sep 10 17:14:38.224621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.224489 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4648ac7e-e79a-4267-87bf-5fd7b87daf7c" containerName="console" Sep 10 17:14:38.224621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.224502 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd67450-0bb9-4894-8e12-02053f31faf1" containerName="registry" Sep 10 17:14:38.224621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.224507 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd67450-0bb9-4894-8e12-02053f31faf1" containerName="registry" Sep 10 17:14:38.224621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.224585 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4648ac7e-e79a-4267-87bf-5fd7b87daf7c" containerName="console" Sep 10 17:14:38.224621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.224596 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd67450-0bb9-4894-8e12-02053f31faf1" containerName="registry" Sep 10 17:14:38.227976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.227869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.243273 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.243241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b4f469b7-r2f5n"] Sep 10 17:14:38.287838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.287805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:38.344062 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.344027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4gt\" (UniqueName: \"kubernetes.io/projected/72a2b0c1-990c-4de3-aad9-4147db89def1-kube-api-access-mz4gt\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.344251 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.344074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-oauth-config\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.344251 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.344146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-serving-cert\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.344251 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.344166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-service-ca\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.344251 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.344198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-trusted-ca-bundle\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.344251 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.344244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-console-config\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.344452 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.344271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-oauth-serving-cert\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.345917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.345867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:38.445325 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.445273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4gt\" (UniqueName: \"kubernetes.io/projected/72a2b0c1-990c-4de3-aad9-4147db89def1-kube-api-access-mz4gt\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.445325 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.445328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-oauth-config\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.445598 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.445374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-serving-cert\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.445598 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.445390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-service-ca\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.445598 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.445406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-trusted-ca-bundle\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.445598 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.445424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-console-config\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.445598 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.445442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-oauth-serving-cert\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.446311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.446281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-service-ca\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.446448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.446308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-oauth-serving-cert\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.446448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.446297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-console-config\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.446448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.446386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-trusted-ca-bundle\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.447829 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.447805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-oauth-config\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.448084 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.448065 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-serving-cert\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.453467 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.453428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4gt\" (UniqueName: \"kubernetes.io/projected/72a2b0c1-990c-4de3-aad9-4147db89def1-kube-api-access-mz4gt\") pod \"console-84b4f469b7-r2f5n\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.538494 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.538398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:38.816831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:38.816800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b4f469b7-r2f5n"] Sep 10 17:14:38.819817 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:14:38.819781 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a2b0c1_990c_4de3_aad9_4147db89def1.slice/crio-a929e0b7fbc8f70258618add76d7800bb08719197f1dc1960bfd43048359f659 WatchSource:0}: Error finding container a929e0b7fbc8f70258618add76d7800bb08719197f1dc1960bfd43048359f659: Status 404 returned error can't find the container with id a929e0b7fbc8f70258618add76d7800bb08719197f1dc1960bfd43048359f659 Sep 10 17:14:39.341402 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:39.341335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b4f469b7-r2f5n" event={"ID":"72a2b0c1-990c-4de3-aad9-4147db89def1","Type":"ContainerStarted","Data":"328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde"} Sep 10 17:14:39.341798 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:39.341457 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b4f469b7-r2f5n" event={"ID":"72a2b0c1-990c-4de3-aad9-4147db89def1","Type":"ContainerStarted","Data":"a929e0b7fbc8f70258618add76d7800bb08719197f1dc1960bfd43048359f659"} Sep 10 17:14:39.361090 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:39.361043 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b4f469b7-r2f5n" podStartSLOduration=1.361030114 podStartE2EDuration="1.361030114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:39.360383507 +0000 UTC m=+185.341463505" watchObservedRunningTime="2026-09-10 17:14:39.361030114 +0000 UTC m=+185.342110111" Sep 10 17:14:48.539755 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:48.539709 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:48.540154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:48.539852 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:48.544761 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:48.544732 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:49.382194 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:49.382165 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:14:49.453508 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:49.453471 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774cd57cb9-5rbf7"] Sep 10 17:14:49.619528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:49.619494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:49.724683 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:49.724590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:14:49.813324 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:14:49.813295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:15:14.475488 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.475450 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-774cd57cb9-5rbf7" podUID="ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" containerName="console" containerID="cri-o://37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17" gracePeriod=15 Sep 10 17:15:14.708403 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.708377 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:15:14.737611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.737531 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-trusted-ca-bundle\") pod \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " Sep 10 17:15:14.737611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.737597 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-config\") pod \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " Sep 10 17:15:14.737851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.737663 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-oauth-serving-cert\") pod \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " Sep 10 17:15:14.737851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.737695 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-serving-cert\") pod \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " Sep 10 17:15:14.737851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.737730 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-oauth-config\") pod \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " Sep 10 17:15:14.737851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.737755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-service-ca\") pod \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " Sep 10 17:15:14.737851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.737802 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s2jd\" (UniqueName: \"kubernetes.io/projected/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-kube-api-access-9s2jd\") pod \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\" (UID: \"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283\") " Sep 10 17:15:14.738263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.738067 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" (UID: "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:15:14.738481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.738446 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-service-ca" (OuterVolumeSpecName: "service-ca") pod "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" (UID: "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:15:14.738481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.738455 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-config" (OuterVolumeSpecName: "console-config") pod "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" (UID: "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:15:14.738627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.738594 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" (UID: "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:15:14.740599 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.740570 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-kube-api-access-9s2jd" (OuterVolumeSpecName: "kube-api-access-9s2jd") pod "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" (UID: "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283"). InnerVolumeSpecName "kube-api-access-9s2jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:15:14.740699 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.740608 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" (UID: "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:15:14.740775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.740752 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" (UID: "ca7f59c3-ba22-41a2-9ef2-c3a20e54c283"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:15:14.839036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.838993 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:15:14.839036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.839030 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-oauth-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:15:14.839036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.839043 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:15:14.839265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.839056 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-console-oauth-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:15:14.839265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.839066 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:15:14.839265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.839074 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2jd\" (UniqueName: \"kubernetes.io/projected/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-kube-api-access-9s2jd\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:15:14.839265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:14.839082 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283-trusted-ca-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:15:15.475171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.475132 2576 generic.go:356] "Generic (PLEG): container finished" podID="ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" containerID="37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17" exitCode=0 Sep 10 17:15:15.475335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.475188 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774cd57cb9-5rbf7" Sep 10 17:15:15.475335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.475194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774cd57cb9-5rbf7" event={"ID":"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283","Type":"ContainerDied","Data":"37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17"} Sep 10 17:15:15.475335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.475317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774cd57cb9-5rbf7" event={"ID":"ca7f59c3-ba22-41a2-9ef2-c3a20e54c283","Type":"ContainerDied","Data":"79caf2ba114ed8d28f7968b533facec89d102d09c9dea5e71d40675866432787"} Sep 10 17:15:15.475445 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.475336 2576 scope.go:117] "RemoveContainer" containerID="37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17" Sep 10 17:15:15.483473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.483453 2576 scope.go:117] "RemoveContainer" containerID="37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17" Sep 10 17:15:15.483823 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:15:15.483726 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17\": container with ID starting with 37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17 not found: ID does not exist" containerID="37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17" Sep 10 17:15:15.483823 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.483754 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17"} err="failed to get container status \"37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17\": rpc error: code = NotFound desc = could not find container \"37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17\": container with ID starting with 37380fdfa3f901ccb3bd6c036aa72c9d31bf3dc82871f703ec0b58c407f63b17 not found: ID does not exist" Sep 10 17:15:15.516335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.516304 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774cd57cb9-5rbf7"] Sep 10 17:15:15.523550 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:15.523523 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-774cd57cb9-5rbf7"] Sep 10 17:15:16.484484 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:16.484452 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" path="/var/lib/kubelet/pods/ca7f59c3-ba22-41a2-9ef2-c3a20e54c283/volumes" Sep 10 17:15:44.719324 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:44.719283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:15:44.821750 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:44.821717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:15:56.772351 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:56.772309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:15:56.849171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:56.849131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:15:58.844928 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:15:58.844888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:03.458513 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:03.458463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:03.568403 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:03.568364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:21.563747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.563720 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64c47bb4df-j78rl"] Sep 10 17:16:21.564205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.564025 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" containerName="console" Sep 10 17:16:21.564205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.564040 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" containerName="console" Sep 10 17:16:21.564205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.564105 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca7f59c3-ba22-41a2-9ef2-c3a20e54c283" containerName="console" Sep 10 17:16:21.565981 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.565959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.582374 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.582348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c47bb4df-j78rl"] Sep 10 17:16:21.632732 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.632703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:21.638073 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.638042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-console-config\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.638201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.638089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-oauth-serving-cert\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.638201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.638107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-serving-cert\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.638201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.638153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-service-ca\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.638201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.638180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-oauth-config\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.638400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.638238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-trusted-ca-bundle\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.638400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.638262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgqn\" (UniqueName: \"kubernetes.io/projected/ea694fba-f496-4702-844f-5eeabda43f3a-kube-api-access-qcgqn\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.712211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.712148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:21.739515 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.739487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-trusted-ca-bundle\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.739670 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.739524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgqn\" (UniqueName: \"kubernetes.io/projected/ea694fba-f496-4702-844f-5eeabda43f3a-kube-api-access-qcgqn\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.739670 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.739592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-console-config\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.739670 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.739642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-oauth-serving-cert\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.739670 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.739666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-serving-cert\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.739902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.739698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-service-ca\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.739902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.739728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-oauth-config\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.740391 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.740367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-oauth-serving-cert\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.740508 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.740433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-service-ca\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.740508 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.740411 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-console-config\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.740591 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.740503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-trusted-ca-bundle\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.742084 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.742058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-oauth-config\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.742187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.742160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-serving-cert\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.750458 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.750435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgqn\" (UniqueName: \"kubernetes.io/projected/ea694fba-f496-4702-844f-5eeabda43f3a-kube-api-access-qcgqn\") pod \"console-64c47bb4df-j78rl\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:21.788172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.788136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:21.877242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:21.877209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:22.150535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:22.150465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c47bb4df-j78rl"] Sep 10 17:16:22.152675 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:16:22.152646 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea694fba_f496_4702_844f_5eeabda43f3a.slice/crio-52e5044248225aff634154cf72a4e67f37fca61d2e02f50bc2814e94f5f3a6a1 WatchSource:0}: Error finding container 52e5044248225aff634154cf72a4e67f37fca61d2e02f50bc2814e94f5f3a6a1: Status 404 returned error can't find the container with id 52e5044248225aff634154cf72a4e67f37fca61d2e02f50bc2814e94f5f3a6a1 Sep 10 17:16:22.389377 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:22.389346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:22.508858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:22.508781 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:22.721373 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:22.721334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c47bb4df-j78rl" event={"ID":"ea694fba-f496-4702-844f-5eeabda43f3a","Type":"ContainerStarted","Data":"04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa"} Sep 10 17:16:22.721781 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:22.721392 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c47bb4df-j78rl" event={"ID":"ea694fba-f496-4702-844f-5eeabda43f3a","Type":"ContainerStarted","Data":"52e5044248225aff634154cf72a4e67f37fca61d2e02f50bc2814e94f5f3a6a1"} Sep 10 17:16:22.741854 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:22.741805 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64c47bb4df-j78rl" podStartSLOduration=1.741788243 podStartE2EDuration="1.741788243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:16:22.740296821 +0000 UTC m=+288.721376820" watchObservedRunningTime="2026-09-10 17:16:22.741788243 +0000 UTC m=+288.722868241" Sep 10 17:16:25.132670 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:25.132630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:25.223887 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:25.223853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:26.293849 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.293815 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm"] Sep 10 17:16:26.296175 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.296160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.298793 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.298773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-96ksv\"" Sep 10 17:16:26.298917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.298773 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 17:16:26.299756 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.299737 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 17:16:26.311229 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.311208 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm"] Sep 10 17:16:26.377377 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.377346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.377519 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.377379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.377519 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.377407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jhc\" (UniqueName: \"kubernetes.io/projected/d4ab2722-40d6-4446-a5a1-aebdb0224af5-kube-api-access-l2jhc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.478742 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.478690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.478742 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.478741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.478742 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.478761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jhc\" (UniqueName: \"kubernetes.io/projected/d4ab2722-40d6-4446-a5a1-aebdb0224af5-kube-api-access-l2jhc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.479143 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.479121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.479222 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.479177 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.488020 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.487996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jhc\" (UniqueName: \"kubernetes.io/projected/d4ab2722-40d6-4446-a5a1-aebdb0224af5-kube-api-access-l2jhc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.604927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.604900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:26.877321 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:26.877247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm"] Sep 10 17:16:26.880096 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:16:26.880058 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ab2722_40d6_4446_a5a1_aebdb0224af5.slice/crio-5b60e07c0205da9c55a93fd3044199dfe464d11ec70b4e74107533faf85ee34b WatchSource:0}: Error finding container 5b60e07c0205da9c55a93fd3044199dfe464d11ec70b4e74107533faf85ee34b: Status 404 returned error can't find the container with id 5b60e07c0205da9c55a93fd3044199dfe464d11ec70b4e74107533faf85ee34b Sep 10 17:16:27.741593 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:27.741559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" event={"ID":"d4ab2722-40d6-4446-a5a1-aebdb0224af5","Type":"ContainerStarted","Data":"5b60e07c0205da9c55a93fd3044199dfe464d11ec70b4e74107533faf85ee34b"} Sep 10 17:16:31.878031 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:31.877982 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:31.878452 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:31.878088 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:31.883467 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:31.883439 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:32.770226 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:32.770195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:16:32.823006 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:32.822972 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b4f469b7-r2f5n"] Sep 10 17:16:32.955210 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:32.955174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:33.055819 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:33.055728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:33.157172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:33.157081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:16:34.404088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:34.404064 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:16:34.404088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:34.404078 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:16:34.420775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:34.420752 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 17:16:34.774560 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:34.774481 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerID="5bfcebdccc0040f0068c77e50f243cd486957e11291646bdfebb8b642d9d4956" exitCode=0 Sep 10 17:16:34.774560 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:34.774554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" event={"ID":"d4ab2722-40d6-4446-a5a1-aebdb0224af5","Type":"ContainerDied","Data":"5bfcebdccc0040f0068c77e50f243cd486957e11291646bdfebb8b642d9d4956"} Sep 10 17:16:48.827910 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:48.827856 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerID="c47f091497c9bf9baaebb062d0864552c3dd39329b782fa8b2c4005b022c3863" exitCode=0 Sep 10 17:16:48.828323 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:48.827942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" event={"ID":"d4ab2722-40d6-4446-a5a1-aebdb0224af5","Type":"ContainerDied","Data":"c47f091497c9bf9baaebb062d0864552c3dd39329b782fa8b2c4005b022c3863"} Sep 10 17:16:48.828982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:48.828969 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:16:55.861730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:55.861698 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerID="2475f8c03c538661b5e0591a79f9a77a796c9886b90e7f55d7c7a0914bfe58ad" exitCode=0 Sep 10 17:16:55.862166 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:55.861755 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" event={"ID":"d4ab2722-40d6-4446-a5a1-aebdb0224af5","Type":"ContainerDied","Data":"2475f8c03c538661b5e0591a79f9a77a796c9886b90e7f55d7c7a0914bfe58ad"} Sep 10 17:16:56.994031 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:56.994006 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:57.138960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.138862 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-bundle\") pod \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " Sep 10 17:16:57.138960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.138935 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-util\") pod \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " Sep 10 17:16:57.138960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.138953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2jhc\" (UniqueName: \"kubernetes.io/projected/d4ab2722-40d6-4446-a5a1-aebdb0224af5-kube-api-access-l2jhc\") pod \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\" (UID: \"d4ab2722-40d6-4446-a5a1-aebdb0224af5\") " Sep 10 17:16:57.139506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.139482 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-bundle" (OuterVolumeSpecName: "bundle") pod "d4ab2722-40d6-4446-a5a1-aebdb0224af5" (UID: "d4ab2722-40d6-4446-a5a1-aebdb0224af5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:16:57.141240 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.141218 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ab2722-40d6-4446-a5a1-aebdb0224af5-kube-api-access-l2jhc" (OuterVolumeSpecName: "kube-api-access-l2jhc") pod "d4ab2722-40d6-4446-a5a1-aebdb0224af5" (UID: "d4ab2722-40d6-4446-a5a1-aebdb0224af5"). InnerVolumeSpecName "kube-api-access-l2jhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:16:57.143799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.143778 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-util" (OuterVolumeSpecName: "util") pod "d4ab2722-40d6-4446-a5a1-aebdb0224af5" (UID: "d4ab2722-40d6-4446-a5a1-aebdb0224af5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:16:57.240343 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.240314 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:57.240343 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.240338 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2jhc\" (UniqueName: \"kubernetes.io/projected/d4ab2722-40d6-4446-a5a1-aebdb0224af5-kube-api-access-l2jhc\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:57.240501 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.240369 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4ab2722-40d6-4446-a5a1-aebdb0224af5-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:57.850942 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.850862 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84b4f469b7-r2f5n" podUID="72a2b0c1-990c-4de3-aad9-4147db89def1" containerName="console" containerID="cri-o://328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde" gracePeriod=15 Sep 10 17:16:57.870747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.870716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" event={"ID":"d4ab2722-40d6-4446-a5a1-aebdb0224af5","Type":"ContainerDied","Data":"5b60e07c0205da9c55a93fd3044199dfe464d11ec70b4e74107533faf85ee34b"} Sep 10 17:16:57.870747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.870740 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rvrm" Sep 10 17:16:57.870958 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:57.870762 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b60e07c0205da9c55a93fd3044199dfe464d11ec70b4e74107533faf85ee34b" Sep 10 17:16:58.093578 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.093554 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:16:58.249402 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249304 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-oauth-serving-cert\") pod \"72a2b0c1-990c-4de3-aad9-4147db89def1\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " Sep 10 17:16:58.249402 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249377 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-trusted-ca-bundle\") pod \"72a2b0c1-990c-4de3-aad9-4147db89def1\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " Sep 10 17:16:58.249629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249417 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz4gt\" (UniqueName: \"kubernetes.io/projected/72a2b0c1-990c-4de3-aad9-4147db89def1-kube-api-access-mz4gt\") pod \"72a2b0c1-990c-4de3-aad9-4147db89def1\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " Sep 10 17:16:58.249629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249443 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-console-config\") pod \"72a2b0c1-990c-4de3-aad9-4147db89def1\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " Sep 10 17:16:58.249629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249485 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-oauth-config\") pod \"72a2b0c1-990c-4de3-aad9-4147db89def1\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " Sep 10 17:16:58.249629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-serving-cert\") pod \"72a2b0c1-990c-4de3-aad9-4147db89def1\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " Sep 10 17:16:58.249629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249530 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-service-ca\") pod \"72a2b0c1-990c-4de3-aad9-4147db89def1\" (UID: \"72a2b0c1-990c-4de3-aad9-4147db89def1\") " Sep 10 17:16:58.249865 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.249807 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "72a2b0c1-990c-4de3-aad9-4147db89def1" (UID: "72a2b0c1-990c-4de3-aad9-4147db89def1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:16:58.250027 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.250002 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "72a2b0c1-990c-4de3-aad9-4147db89def1" (UID: "72a2b0c1-990c-4de3-aad9-4147db89def1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:16:58.250097 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.250064 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-service-ca" (OuterVolumeSpecName: "service-ca") pod "72a2b0c1-990c-4de3-aad9-4147db89def1" (UID: "72a2b0c1-990c-4de3-aad9-4147db89def1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:16:58.250181 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.250156 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-console-config" (OuterVolumeSpecName: "console-config") pod "72a2b0c1-990c-4de3-aad9-4147db89def1" (UID: "72a2b0c1-990c-4de3-aad9-4147db89def1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:16:58.251695 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.251670 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "72a2b0c1-990c-4de3-aad9-4147db89def1" (UID: "72a2b0c1-990c-4de3-aad9-4147db89def1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:16:58.251695 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.251684 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a2b0c1-990c-4de3-aad9-4147db89def1-kube-api-access-mz4gt" (OuterVolumeSpecName: "kube-api-access-mz4gt") pod "72a2b0c1-990c-4de3-aad9-4147db89def1" (UID: "72a2b0c1-990c-4de3-aad9-4147db89def1"). InnerVolumeSpecName "kube-api-access-mz4gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:16:58.251695 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.251690 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "72a2b0c1-990c-4de3-aad9-4147db89def1" (UID: "72a2b0c1-990c-4de3-aad9-4147db89def1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:16:58.350361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.350326 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-oauth-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:58.350361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.350355 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a2b0c1-990c-4de3-aad9-4147db89def1-console-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:58.350361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.350365 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:58.350580 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.350374 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-oauth-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:58.350580 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.350383 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-trusted-ca-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:58.350580 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.350392 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4gt\" (UniqueName: \"kubernetes.io/projected/72a2b0c1-990c-4de3-aad9-4147db89def1-kube-api-access-mz4gt\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:58.350580 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.350402 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a2b0c1-990c-4de3-aad9-4147db89def1-console-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:16:58.876014 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.875977 2576 generic.go:356] "Generic (PLEG): container finished" podID="72a2b0c1-990c-4de3-aad9-4147db89def1" containerID="328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde" exitCode=0 Sep 10 17:16:58.876189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.876028 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b4f469b7-r2f5n" Sep 10 17:16:58.876189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.876068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b4f469b7-r2f5n" event={"ID":"72a2b0c1-990c-4de3-aad9-4147db89def1","Type":"ContainerDied","Data":"328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde"} Sep 10 17:16:58.876189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.876117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b4f469b7-r2f5n" event={"ID":"72a2b0c1-990c-4de3-aad9-4147db89def1","Type":"ContainerDied","Data":"a929e0b7fbc8f70258618add76d7800bb08719197f1dc1960bfd43048359f659"} Sep 10 17:16:58.876189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.876137 2576 scope.go:117] "RemoveContainer" containerID="328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde" Sep 10 17:16:58.888062 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.888029 2576 scope.go:117] "RemoveContainer" containerID="328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde" Sep 10 17:16:58.888522 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:16:58.888502 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde\": container with ID starting with 328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde not found: ID does not exist" containerID="328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde" Sep 10 17:16:58.888610 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.888529 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde"} err="failed to get container status \"328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde\": rpc error: code = NotFound desc = could not find container \"328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde\": container with ID starting with 328a572261cb6a2a1d5353ccab8f8f146c395f343a9b402a2b9954fa3d034cde not found: ID does not exist" Sep 10 17:16:58.898546 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.898520 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b4f469b7-r2f5n"] Sep 10 17:16:58.904179 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:16:58.904156 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b4f469b7-r2f5n"] Sep 10 17:17:00.484752 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:00.484712 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a2b0c1-990c-4de3-aad9-4147db89def1" path="/var/lib/kubelet/pods/72a2b0c1-990c-4de3-aad9-4147db89def1/volumes" Sep 10 17:17:02.778535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:02.778498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:04.063682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:04.063644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:06.797052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:06.797012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:06.902608 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:06.902574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:07.710154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:07.710103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:08.831299 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:08.831245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:08.964072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:08.964033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:09.246735 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.246644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:09.373071 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.373043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:09.528140 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528064 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht"] Sep 10 17:17:09.528347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528336 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerName="util" Sep 10 17:17:09.528389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528349 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerName="util" Sep 10 17:17:09.528389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528362 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerName="extract" Sep 10 17:17:09.528389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528368 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerName="extract" Sep 10 17:17:09.528389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528375 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a2b0c1-990c-4de3-aad9-4147db89def1" containerName="console" Sep 10 17:17:09.528389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528380 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a2b0c1-990c-4de3-aad9-4147db89def1" containerName="console" Sep 10 17:17:09.528389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528387 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerName="pull" Sep 10 17:17:09.528591 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528392 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerName="pull" Sep 10 17:17:09.528591 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528438 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a2b0c1-990c-4de3-aad9-4147db89def1" containerName="console" Sep 10 17:17:09.528591 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.528448 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ab2722-40d6-4446-a5a1-aebdb0224af5" containerName="extract" Sep 10 17:17:09.532621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.532603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.535086 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.535064 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 17:17:09.536079 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.536049 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-96ksv\"" Sep 10 17:17:09.536183 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.536083 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 17:17:09.543266 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.543243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht"] Sep 10 17:17:09.631408 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.631375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.631582 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.631424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.631582 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.631454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk75t\" (UniqueName: \"kubernetes.io/projected/23247bd1-28bf-4ff0-ac9e-727f6c366c67-kube-api-access-kk75t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.732172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.732134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.732329 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.732180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.732329 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.732203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk75t\" (UniqueName: \"kubernetes.io/projected/23247bd1-28bf-4ff0-ac9e-727f6c366c67-kube-api-access-kk75t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.732602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.732578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.732640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.732586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.744393 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.744370 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk75t\" (UniqueName: \"kubernetes.io/projected/23247bd1-28bf-4ff0-ac9e-727f6c366c67-kube-api-access-kk75t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:09.841633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:09.841599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:10.149438 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.149414 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht"] Sep 10 17:17:10.151495 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:17:10.151467 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23247bd1_28bf_4ff0_ac9e_727f6c366c67.slice/crio-a6a6a4558a3cfa303c0e5c27d577b071e4c65280132fdf87dbc2fdbefc452ffd WatchSource:0}: Error finding container a6a6a4558a3cfa303c0e5c27d577b071e4c65280132fdf87dbc2fdbefc452ffd: Status 404 returned error can't find the container with id a6a6a4558a3cfa303c0e5c27d577b071e4c65280132fdf87dbc2fdbefc452ffd Sep 10 17:17:10.180378 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.180350 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trqnh"] Sep 10 17:17:10.183518 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.183502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.187236 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.187219 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wk8lj\"" Sep 10 17:17:10.187900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.187867 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 17:17:10.188159 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.188143 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 17:17:10.214705 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.214677 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trqnh"] Sep 10 17:17:10.236347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.236323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c49809-b1cc-4ce9-b55b-eeb15ad20124-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trqnh\" (UID: \"86c49809-b1cc-4ce9-b55b-eeb15ad20124\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.236481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.236357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkr9\" (UniqueName: \"kubernetes.io/projected/86c49809-b1cc-4ce9-b55b-eeb15ad20124-kube-api-access-zgkr9\") pod \"cert-manager-webhook-6486ffd997-trqnh\" (UID: \"86c49809-b1cc-4ce9-b55b-eeb15ad20124\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.337305 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.337271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c49809-b1cc-4ce9-b55b-eeb15ad20124-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trqnh\" (UID: \"86c49809-b1cc-4ce9-b55b-eeb15ad20124\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.337473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.337315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkr9\" (UniqueName: \"kubernetes.io/projected/86c49809-b1cc-4ce9-b55b-eeb15ad20124-kube-api-access-zgkr9\") pod \"cert-manager-webhook-6486ffd997-trqnh\" (UID: \"86c49809-b1cc-4ce9-b55b-eeb15ad20124\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.346163 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.346129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c49809-b1cc-4ce9-b55b-eeb15ad20124-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trqnh\" (UID: \"86c49809-b1cc-4ce9-b55b-eeb15ad20124\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.346276 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.346184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkr9\" (UniqueName: \"kubernetes.io/projected/86c49809-b1cc-4ce9-b55b-eeb15ad20124-kube-api-access-zgkr9\") pod \"cert-manager-webhook-6486ffd997-trqnh\" (UID: \"86c49809-b1cc-4ce9-b55b-eeb15ad20124\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.512263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.512174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:10.779935 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.779911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trqnh"] Sep 10 17:17:10.781945 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:17:10.781916 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86c49809_b1cc_4ce9_b55b_eeb15ad20124.slice/crio-31fbe8d603caa4dbfb21763fc60dc73bf93d179cc207ff72b617566da29d6ccf WatchSource:0}: Error finding container 31fbe8d603caa4dbfb21763fc60dc73bf93d179cc207ff72b617566da29d6ccf: Status 404 returned error can't find the container with id 31fbe8d603caa4dbfb21763fc60dc73bf93d179cc207ff72b617566da29d6ccf Sep 10 17:17:10.923072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.923040 2576 generic.go:356] "Generic (PLEG): container finished" podID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerID="c78821b84515e0743d6ae647406056e4298e9dfc3f8cfb17ec59d4e0594f00d7" exitCode=0 Sep 10 17:17:10.923531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.923130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" event={"ID":"23247bd1-28bf-4ff0-ac9e-727f6c366c67","Type":"ContainerDied","Data":"c78821b84515e0743d6ae647406056e4298e9dfc3f8cfb17ec59d4e0594f00d7"} Sep 10 17:17:10.923531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.923188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" event={"ID":"23247bd1-28bf-4ff0-ac9e-727f6c366c67","Type":"ContainerStarted","Data":"a6a6a4558a3cfa303c0e5c27d577b071e4c65280132fdf87dbc2fdbefc452ffd"} Sep 10 17:17:10.924460 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:10.924431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" event={"ID":"86c49809-b1cc-4ce9-b55b-eeb15ad20124","Type":"ContainerStarted","Data":"31fbe8d603caa4dbfb21763fc60dc73bf93d179cc207ff72b617566da29d6ccf"} Sep 10 17:17:11.601636 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:11.601593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:11.787693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:11.787657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:13.938594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:13.938555 2576 generic.go:356] "Generic (PLEG): container finished" podID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerID="b9d8d8f189936e8dd8d70c3eafdbbd028238991a4541325261bf546bfd89988d" exitCode=0 Sep 10 17:17:13.939095 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:13.938650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" event={"ID":"23247bd1-28bf-4ff0-ac9e-727f6c366c67","Type":"ContainerDied","Data":"b9d8d8f189936e8dd8d70c3eafdbbd028238991a4541325261bf546bfd89988d"} Sep 10 17:17:14.023476 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:14.023433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:14.123331 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:14.123295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:14.944290 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:14.944259 2576 generic.go:356] "Generic (PLEG): container finished" podID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerID="9c9c984c356c33db44abfbd68de5c6a7ca6119d36a0a1f081cabd27379881682" exitCode=0 Sep 10 17:17:14.944721 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:14.944305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" event={"ID":"23247bd1-28bf-4ff0-ac9e-727f6c366c67","Type":"ContainerDied","Data":"9c9c984c356c33db44abfbd68de5c6a7ca6119d36a0a1f081cabd27379881682"} Sep 10 17:17:16.092461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.092421 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:16.182136 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.182100 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-bundle\") pod \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " Sep 10 17:17:16.182332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.182169 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk75t\" (UniqueName: \"kubernetes.io/projected/23247bd1-28bf-4ff0-ac9e-727f6c366c67-kube-api-access-kk75t\") pod \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " Sep 10 17:17:16.182332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.182211 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-util\") pod \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\" (UID: \"23247bd1-28bf-4ff0-ac9e-727f6c366c67\") " Sep 10 17:17:16.182677 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.182644 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-bundle" (OuterVolumeSpecName: "bundle") pod "23247bd1-28bf-4ff0-ac9e-727f6c366c67" (UID: "23247bd1-28bf-4ff0-ac9e-727f6c366c67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:17:16.184628 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.184600 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23247bd1-28bf-4ff0-ac9e-727f6c366c67-kube-api-access-kk75t" (OuterVolumeSpecName: "kube-api-access-kk75t") pod "23247bd1-28bf-4ff0-ac9e-727f6c366c67" (UID: "23247bd1-28bf-4ff0-ac9e-727f6c366c67"). InnerVolumeSpecName "kube-api-access-kk75t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:17:16.188637 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.188613 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-util" (OuterVolumeSpecName: "util") pod "23247bd1-28bf-4ff0-ac9e-727f6c366c67" (UID: "23247bd1-28bf-4ff0-ac9e-727f6c366c67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:17:16.283037 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.282960 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:16.283037 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.282990 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk75t\" (UniqueName: \"kubernetes.io/projected/23247bd1-28bf-4ff0-ac9e-727f6c366c67-kube-api-access-kk75t\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:16.283037 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.283003 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23247bd1-28bf-4ff0-ac9e-727f6c366c67-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:16.954123 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.954087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" event={"ID":"23247bd1-28bf-4ff0-ac9e-727f6c366c67","Type":"ContainerDied","Data":"a6a6a4558a3cfa303c0e5c27d577b071e4c65280132fdf87dbc2fdbefc452ffd"} Sep 10 17:17:16.954123 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.954118 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftmkht" Sep 10 17:17:16.954374 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:16.954143 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6a6a4558a3cfa303c0e5c27d577b071e4c65280132fdf87dbc2fdbefc452ffd" Sep 10 17:17:17.960627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:17.960593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" event={"ID":"86c49809-b1cc-4ce9-b55b-eeb15ad20124","Type":"ContainerStarted","Data":"d2c6295574e80c8c40c97c116ef71d7b3b68838223d421e52f74e6a4b03024fc"} Sep 10 17:17:17.961028 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:17.960713 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:17.979770 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:17.979719 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" podStartSLOduration=1.398389513 podStartE2EDuration="7.979704094s" totalImagesPullingTime="6.581314581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:17:17.978129692 +0000 UTC m=+343.959209690" watchObservedRunningTime="2026-09-10 17:17:17.979704094 +0000 UTC m=+343.960784089" Sep 10 17:17:19.393439 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.393405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:19.900931 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.900893 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mggzf"] Sep 10 17:17:19.901190 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.901178 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerName="extract" Sep 10 17:17:19.901237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.901192 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerName="extract" Sep 10 17:17:19.901237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.901213 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerName="pull" Sep 10 17:17:19.901237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.901219 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerName="pull" Sep 10 17:17:19.901237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.901225 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerName="util" Sep 10 17:17:19.901237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.901231 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerName="util" Sep 10 17:17:19.901385 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.901275 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23247bd1-28bf-4ff0-ac9e-727f6c366c67" containerName="extract" Sep 10 17:17:19.904223 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.904206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:19.906478 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.906452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zmpts\"" Sep 10 17:17:19.915056 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:19.915033 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mggzf"] Sep 10 17:17:20.010612 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.010578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9dmr\" (UniqueName: \"kubernetes.io/projected/a22617af-7b3e-4550-85b7-48aa058ec1ed-kube-api-access-r9dmr\") pod \"cert-manager-55b4d954d5-mggzf\" (UID: \"a22617af-7b3e-4550-85b7-48aa058ec1ed\") " pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:20.010739 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.010626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22617af-7b3e-4550-85b7-48aa058ec1ed-bound-sa-token\") pod \"cert-manager-55b4d954d5-mggzf\" (UID: \"a22617af-7b3e-4550-85b7-48aa058ec1ed\") " pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:20.111706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.111672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22617af-7b3e-4550-85b7-48aa058ec1ed-bound-sa-token\") pod \"cert-manager-55b4d954d5-mggzf\" (UID: \"a22617af-7b3e-4550-85b7-48aa058ec1ed\") " pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:20.111857 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.111731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9dmr\" (UniqueName: \"kubernetes.io/projected/a22617af-7b3e-4550-85b7-48aa058ec1ed-kube-api-access-r9dmr\") pod \"cert-manager-55b4d954d5-mggzf\" (UID: \"a22617af-7b3e-4550-85b7-48aa058ec1ed\") " pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:20.120203 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.120171 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22617af-7b3e-4550-85b7-48aa058ec1ed-bound-sa-token\") pod \"cert-manager-55b4d954d5-mggzf\" (UID: \"a22617af-7b3e-4550-85b7-48aa058ec1ed\") " pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:20.120317 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.120233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9dmr\" (UniqueName: \"kubernetes.io/projected/a22617af-7b3e-4550-85b7-48aa058ec1ed-kube-api-access-r9dmr\") pod \"cert-manager-55b4d954d5-mggzf\" (UID: \"a22617af-7b3e-4550-85b7-48aa058ec1ed\") " pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:20.212534 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.212451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mggzf" Sep 10 17:17:20.484979 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:17:20.484946 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22617af_7b3e_4550_85b7_48aa058ec1ed.slice/crio-010b1ab7574a4884fca301695852a6e475366534837c318c5755f36ba546d938 WatchSource:0}: Error finding container 010b1ab7574a4884fca301695852a6e475366534837c318c5755f36ba546d938: Status 404 returned error can't find the container with id 010b1ab7574a4884fca301695852a6e475366534837c318c5755f36ba546d938 Sep 10 17:17:20.486711 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.486692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mggzf"] Sep 10 17:17:20.973497 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.973464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mggzf" event={"ID":"a22617af-7b3e-4550-85b7-48aa058ec1ed","Type":"ContainerStarted","Data":"99ad309373ca389c7b7ac41166d2a06be8645cfbe359217eabdb36c30b30bd66"} Sep 10 17:17:20.973497 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:20.973505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mggzf" event={"ID":"a22617af-7b3e-4550-85b7-48aa058ec1ed","Type":"ContainerStarted","Data":"010b1ab7574a4884fca301695852a6e475366534837c318c5755f36ba546d938"} Sep 10 17:17:21.002086 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:21.002037 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mggzf" podStartSLOduration=2.002020105 podStartE2EDuration="2.002020105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:17:21.000741095 +0000 UTC m=+346.981821123" watchObservedRunningTime="2026-09-10 17:17:21.002020105 +0000 UTC m=+346.983100104" Sep 10 17:17:21.026852 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:21.026819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:21.126142 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:21.126110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:23.966968 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:23.966932 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-trqnh" Sep 10 17:17:27.121767 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:27.121686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:27.227728 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:27.227693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:28.828695 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:28.828657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:30.593010 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.592974 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm"] Sep 10 17:17:30.595315 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.595291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.598105 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.598084 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-96ksv\"" Sep 10 17:17:30.598200 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.598084 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 17:17:30.599266 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.599249 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 17:17:30.605441 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.605421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm"] Sep 10 17:17:30.691672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.691635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmt8\" (UniqueName: \"kubernetes.io/projected/8a3a7c98-f5dd-492f-8c82-0f77a3771151-kube-api-access-sqmt8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.691875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.691699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.691875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.691810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.793073 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.793024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.793073 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.793076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmt8\" (UniqueName: \"kubernetes.io/projected/8a3a7c98-f5dd-492f-8c82-0f77a3771151-kube-api-access-sqmt8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.793331 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.793129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.793442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.793419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.793502 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.793433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.802904 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.802854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmt8\" (UniqueName: \"kubernetes.io/projected/8a3a7c98-f5dd-492f-8c82-0f77a3771151-kube-api-access-sqmt8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:30.905306 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:30.905199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:31.176606 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:31.176578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm"] Sep 10 17:17:31.178668 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:17:31.178639 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3a7c98_f5dd_492f_8c82_0f77a3771151.slice/crio-83a65c9041869cb938c9e51972ac08129562a21c9f1353c828061a67c75fb953 WatchSource:0}: Error finding container 83a65c9041869cb938c9e51972ac08129562a21c9f1353c828061a67c75fb953: Status 404 returned error can't find the container with id 83a65c9041869cb938c9e51972ac08129562a21c9f1353c828061a67c75fb953 Sep 10 17:17:32.021146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:32.021109 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerID="1d63d75be6804cf2e904590377377e0e56caf56aaa4397c626194833e405ed51" exitCode=0 Sep 10 17:17:32.021524 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:32.021179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" event={"ID":"8a3a7c98-f5dd-492f-8c82-0f77a3771151","Type":"ContainerDied","Data":"1d63d75be6804cf2e904590377377e0e56caf56aaa4397c626194833e405ed51"} Sep 10 17:17:32.021524 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:32.021215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" event={"ID":"8a3a7c98-f5dd-492f-8c82-0f77a3771151","Type":"ContainerStarted","Data":"83a65c9041869cb938c9e51972ac08129562a21c9f1353c828061a67c75fb953"} Sep 10 17:17:33.027840 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:33.027800 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerID="12e64b6d1f8ee75b784e612c062ea4c9152ffc457b0adcd15d2b1e11e756657b" exitCode=0 Sep 10 17:17:33.028240 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:33.027900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" event={"ID":"8a3a7c98-f5dd-492f-8c82-0f77a3771151","Type":"ContainerDied","Data":"12e64b6d1f8ee75b784e612c062ea4c9152ffc457b0adcd15d2b1e11e756657b"} Sep 10 17:17:34.034819 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:34.034785 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerID="2ef0162aaccdc8b9d66cd8e5957c004fc60a5e57e800c7daa1411bfd3de38cf8" exitCode=0 Sep 10 17:17:34.035223 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:34.034865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" event={"ID":"8a3a7c98-f5dd-492f-8c82-0f77a3771151","Type":"ContainerDied","Data":"2ef0162aaccdc8b9d66cd8e5957c004fc60a5e57e800c7daa1411bfd3de38cf8"} Sep 10 17:17:35.161561 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.161536 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:35.229989 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.229957 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-bundle\") pod \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " Sep 10 17:17:35.230178 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.230000 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-util\") pod \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " Sep 10 17:17:35.230178 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.230030 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqmt8\" (UniqueName: \"kubernetes.io/projected/8a3a7c98-f5dd-492f-8c82-0f77a3771151-kube-api-access-sqmt8\") pod \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\" (UID: \"8a3a7c98-f5dd-492f-8c82-0f77a3771151\") " Sep 10 17:17:35.230702 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.230669 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-bundle" (OuterVolumeSpecName: "bundle") pod "8a3a7c98-f5dd-492f-8c82-0f77a3771151" (UID: "8a3a7c98-f5dd-492f-8c82-0f77a3771151"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:17:35.232228 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.232197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3a7c98-f5dd-492f-8c82-0f77a3771151-kube-api-access-sqmt8" (OuterVolumeSpecName: "kube-api-access-sqmt8") pod "8a3a7c98-f5dd-492f-8c82-0f77a3771151" (UID: "8a3a7c98-f5dd-492f-8c82-0f77a3771151"). InnerVolumeSpecName "kube-api-access-sqmt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:17:35.236034 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.236012 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-util" (OuterVolumeSpecName: "util") pod "8a3a7c98-f5dd-492f-8c82-0f77a3771151" (UID: "8a3a7c98-f5dd-492f-8c82-0f77a3771151"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:17:35.330672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.330636 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:35.330672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.330667 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7c98-f5dd-492f-8c82-0f77a3771151-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:35.330672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:35.330677 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqmt8\" (UniqueName: \"kubernetes.io/projected/8a3a7c98-f5dd-492f-8c82-0f77a3771151-kube-api-access-sqmt8\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:36.045618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:36.045585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" event={"ID":"8a3a7c98-f5dd-492f-8c82-0f77a3771151","Type":"ContainerDied","Data":"83a65c9041869cb938c9e51972ac08129562a21c9f1353c828061a67c75fb953"} Sep 10 17:17:36.045618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:36.045614 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgt2rm" Sep 10 17:17:36.045836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:36.045631 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83a65c9041869cb938c9e51972ac08129562a21c9f1353c828061a67c75fb953" Sep 10 17:17:39.890300 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890264 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj"] Sep 10 17:17:39.890792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890575 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerName="extract" Sep 10 17:17:39.890792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890588 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerName="extract" Sep 10 17:17:39.890792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890601 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerName="pull" Sep 10 17:17:39.890792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890610 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerName="pull" Sep 10 17:17:39.890792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890626 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerName="util" Sep 10 17:17:39.890792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890632 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerName="util" Sep 10 17:17:39.890792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.890706 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3a7c98-f5dd-492f-8c82-0f77a3771151" containerName="extract" Sep 10 17:17:39.893116 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.893097 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:39.896130 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.896102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 17:17:39.897313 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.897295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-96ksv\"" Sep 10 17:17:39.897434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.897399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 17:17:39.921603 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.921574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj"] Sep 10 17:17:39.966655 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.966609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfdm\" (UniqueName: \"kubernetes.io/projected/d79c0523-92d4-487e-ab06-8acfd54448e0-kube-api-access-bkfdm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:39.966847 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.966666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:39.966847 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:39.966770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.068061 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.068013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.068261 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.068079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfdm\" (UniqueName: \"kubernetes.io/projected/d79c0523-92d4-487e-ab06-8acfd54448e0-kube-api-access-bkfdm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.068261 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.068107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.068419 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.068368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.068486 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.068458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.084451 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.084420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfdm\" (UniqueName: \"kubernetes.io/projected/d79c0523-92d4-487e-ab06-8acfd54448e0-kube-api-access-bkfdm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.202067 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.201975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:40.476656 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.476632 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj"] Sep 10 17:17:40.478551 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:17:40.478526 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79c0523_92d4_487e_ab06_8acfd54448e0.slice/crio-c22c54ff6d644eccddd5213dff59afa73d2ec1a0b52e40c0d1b1553f2c049044 WatchSource:0}: Error finding container c22c54ff6d644eccddd5213dff59afa73d2ec1a0b52e40c0d1b1553f2c049044: Status 404 returned error can't find the container with id c22c54ff6d644eccddd5213dff59afa73d2ec1a0b52e40c0d1b1553f2c049044 Sep 10 17:17:40.748578 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.748490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:40.876319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:40.876282 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:41.066674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:41.066637 2576 generic.go:356] "Generic (PLEG): container finished" podID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerID="081fa8904663c5252b41710b9f3facb6c263c9d036bda0972ba9370c26c5cf7b" exitCode=0 Sep 10 17:17:41.067135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:41.066707 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" event={"ID":"d79c0523-92d4-487e-ab06-8acfd54448e0","Type":"ContainerDied","Data":"081fa8904663c5252b41710b9f3facb6c263c9d036bda0972ba9370c26c5cf7b"} Sep 10 17:17:41.067135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:41.066751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" event={"ID":"d79c0523-92d4-487e-ab06-8acfd54448e0","Type":"ContainerStarted","Data":"c22c54ff6d644eccddd5213dff59afa73d2ec1a0b52e40c0d1b1553f2c049044"} Sep 10 17:17:41.430960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:41.430820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:41.870341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:41.870302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:41.974896 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:41.974793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:42.031076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.031042 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr"] Sep 10 17:17:42.034225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.034210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.037780 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.037757 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 17:17:42.037915 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.037856 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 17:17:42.038127 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.038110 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 17:17:42.038367 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.038342 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 17:17:42.038507 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.038493 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tv4fm\"" Sep 10 17:17:42.059142 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.059120 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr"] Sep 10 17:17:42.073653 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.073623 2576 generic.go:356] "Generic (PLEG): container finished" podID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerID="640e790cde6e70557fcbcc987832d8e8a61427c838d2f5a50aa9179aae4cca89" exitCode=0 Sep 10 17:17:42.074083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.073668 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" event={"ID":"d79c0523-92d4-487e-ab06-8acfd54448e0","Type":"ContainerDied","Data":"640e790cde6e70557fcbcc987832d8e8a61427c838d2f5a50aa9179aae4cca89"} Sep 10 17:17:42.084861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.084839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78a67348-ba67-4210-a24f-e756e5eb8f40-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.084992 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.084905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78a67348-ba67-4210-a24f-e756e5eb8f40-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.084992 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.084964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnj6w\" (UniqueName: \"kubernetes.io/projected/78a67348-ba67-4210-a24f-e756e5eb8f40-kube-api-access-hnj6w\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.085076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.085030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78a67348-ba67-4210-a24f-e756e5eb8f40-tmp\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.085111 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.085087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78a67348-ba67-4210-a24f-e756e5eb8f40-manifests\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.185659 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.185629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78a67348-ba67-4210-a24f-e756e5eb8f40-manifests\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.185800 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.185691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78a67348-ba67-4210-a24f-e756e5eb8f40-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.185800 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.185742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78a67348-ba67-4210-a24f-e756e5eb8f40-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.185930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.185897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnj6w\" (UniqueName: \"kubernetes.io/projected/78a67348-ba67-4210-a24f-e756e5eb8f40-kube-api-access-hnj6w\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.185992 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.185946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78a67348-ba67-4210-a24f-e756e5eb8f40-tmp\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.186319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.186083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78a67348-ba67-4210-a24f-e756e5eb8f40-manifests\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.188348 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.188325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78a67348-ba67-4210-a24f-e756e5eb8f40-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.188450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.188385 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78a67348-ba67-4210-a24f-e756e5eb8f40-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.188450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.188424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78a67348-ba67-4210-a24f-e756e5eb8f40-tmp\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.212517 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.212490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnj6w\" (UniqueName: \"kubernetes.io/projected/78a67348-ba67-4210-a24f-e756e5eb8f40-kube-api-access-hnj6w\") pod \"opendatahub-operator-controller-manager-86f7548996-m2jwr\" (UID: \"78a67348-ba67-4210-a24f-e756e5eb8f40\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.343986 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.343948 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:42.401463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.401439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:42.522647 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.522565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:42.617265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.617226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:42.622837 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.622814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr"] Sep 10 17:17:42.625023 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:17:42.624999 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a67348_ba67_4210_a24f_e756e5eb8f40.slice/crio-8d4af5e3f389aeeaf593c8f4b915d240dcd796e13b1110233ac78cd6b8bcce0f WatchSource:0}: Error finding container 8d4af5e3f389aeeaf593c8f4b915d240dcd796e13b1110233ac78cd6b8bcce0f: Status 404 returned error can't find the container with id 8d4af5e3f389aeeaf593c8f4b915d240dcd796e13b1110233ac78cd6b8bcce0f Sep 10 17:17:42.716725 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.716691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:42.817787 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.817756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:42.919369 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:42.919326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:43.080017 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:43.079930 2576 generic.go:356] "Generic (PLEG): container finished" podID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerID="acc6915e2a29a7fb37177a7cd55b65a2909fb908f81cb609aef7d8b47714d85e" exitCode=0 Sep 10 17:17:43.080422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:43.080021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" event={"ID":"d79c0523-92d4-487e-ab06-8acfd54448e0","Type":"ContainerDied","Data":"acc6915e2a29a7fb37177a7cd55b65a2909fb908f81cb609aef7d8b47714d85e"} Sep 10 17:17:43.081869 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:43.081844 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" event={"ID":"78a67348-ba67-4210-a24f-e756e5eb8f40","Type":"ContainerStarted","Data":"8d4af5e3f389aeeaf593c8f4b915d240dcd796e13b1110233ac78cd6b8bcce0f"} Sep 10 17:17:43.711626 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:43.711586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:43.805026 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:43.804969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:45.661927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.661868 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:45.718928 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.718908 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-util\") pod \"d79c0523-92d4-487e-ab06-8acfd54448e0\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " Sep 10 17:17:45.719025 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.718939 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-bundle\") pod \"d79c0523-92d4-487e-ab06-8acfd54448e0\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " Sep 10 17:17:45.719080 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.719034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkfdm\" (UniqueName: \"kubernetes.io/projected/d79c0523-92d4-487e-ab06-8acfd54448e0-kube-api-access-bkfdm\") pod \"d79c0523-92d4-487e-ab06-8acfd54448e0\" (UID: \"d79c0523-92d4-487e-ab06-8acfd54448e0\") " Sep 10 17:17:45.719821 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.719795 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-bundle" (OuterVolumeSpecName: "bundle") pod "d79c0523-92d4-487e-ab06-8acfd54448e0" (UID: "d79c0523-92d4-487e-ab06-8acfd54448e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:17:45.720981 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.720957 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79c0523-92d4-487e-ab06-8acfd54448e0-kube-api-access-bkfdm" (OuterVolumeSpecName: "kube-api-access-bkfdm") pod "d79c0523-92d4-487e-ab06-8acfd54448e0" (UID: "d79c0523-92d4-487e-ab06-8acfd54448e0"). InnerVolumeSpecName "kube-api-access-bkfdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:17:45.764827 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.764804 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-util" (OuterVolumeSpecName: "util") pod "d79c0523-92d4-487e-ab06-8acfd54448e0" (UID: "d79c0523-92d4-487e-ab06-8acfd54448e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:17:45.820312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.820289 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfdm\" (UniqueName: \"kubernetes.io/projected/d79c0523-92d4-487e-ab06-8acfd54448e0-kube-api-access-bkfdm\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:45.820312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.820313 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:45.820429 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:45.820323 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d79c0523-92d4-487e-ab06-8acfd54448e0-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:17:46.099681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:46.099659 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" Sep 10 17:17:46.099863 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:46.099660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p86mj" event={"ID":"d79c0523-92d4-487e-ab06-8acfd54448e0","Type":"ContainerDied","Data":"c22c54ff6d644eccddd5213dff59afa73d2ec1a0b52e40c0d1b1553f2c049044"} Sep 10 17:17:46.099863 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:46.099774 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c22c54ff6d644eccddd5213dff59afa73d2ec1a0b52e40c0d1b1553f2c049044" Sep 10 17:17:46.101229 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:46.101202 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" event={"ID":"78a67348-ba67-4210-a24f-e756e5eb8f40","Type":"ContainerStarted","Data":"e573b86d21106b216e656fcce572f0f55c30f241b6f2e2ce31da3c8c7c387a43"} Sep 10 17:17:47.107067 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:47.107031 2576 generic.go:356] "Generic (PLEG): container finished" podID="78a67348-ba67-4210-a24f-e756e5eb8f40" containerID="e573b86d21106b216e656fcce572f0f55c30f241b6f2e2ce31da3c8c7c387a43" exitCode=0 Sep 10 17:17:47.107451 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:47.107078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" event={"ID":"78a67348-ba67-4210-a24f-e756e5eb8f40","Type":"ContainerDied","Data":"e573b86d21106b216e656fcce572f0f55c30f241b6f2e2ce31da3c8c7c387a43"} Sep 10 17:17:48.113751 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:48.113718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" event={"ID":"78a67348-ba67-4210-a24f-e756e5eb8f40","Type":"ContainerStarted","Data":"6debf60ad538be7819c67e3099e6cb6a6d714c304bf436a8f780045cbc0925cd"} Sep 10 17:17:48.114145 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:48.113858 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:17:48.159898 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:48.159840 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" podStartSLOduration=2.782558427 podStartE2EDuration="7.159828571s" totalImagesPullingTime="3.363338676s" totalInitContainerRuntime="1.013931468s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:17:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:17:48.158631421 +0000 UTC m=+374.139711432" watchObservedRunningTime="2026-09-10 17:17:48.159828571 +0000 UTC m=+374.140908569" Sep 10 17:17:56.702065 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:56.702030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:56.811255 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:56.811224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:17:59.120337 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:17:59.120309 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-m2jwr" Sep 10 17:18:00.754763 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.754731 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph"] Sep 10 17:18:00.755128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.755043 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerName="util" Sep 10 17:18:00.755128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.755055 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerName="util" Sep 10 17:18:00.755128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.755069 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerName="extract" Sep 10 17:18:00.755128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.755074 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerName="extract" Sep 10 17:18:00.755128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.755085 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerName="pull" Sep 10 17:18:00.755128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.755090 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerName="pull" Sep 10 17:18:00.755353 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.755139 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79c0523-92d4-487e-ab06-8acfd54448e0" containerName="extract" Sep 10 17:18:00.757874 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.757858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.762083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.762064 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 17:18:00.763188 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.763165 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-25vrv\"" Sep 10 17:18:00.763298 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.763189 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 17:18:00.763298 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.763201 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 17:18:00.763298 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.763235 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 17:18:00.763298 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.763171 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:18:00.775641 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.775621 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph"] Sep 10 17:18:00.830266 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.830231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c44f1e05-226d-40c9-8626-484e90514cbb-manager-config\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.830370 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.830292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c44f1e05-226d-40c9-8626-484e90514cbb-metrics-cert\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.830370 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.830357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c44f1e05-226d-40c9-8626-484e90514cbb-cert\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.830456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.830412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptghk\" (UniqueName: \"kubernetes.io/projected/c44f1e05-226d-40c9-8626-484e90514cbb-kube-api-access-ptghk\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.931236 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.931205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptghk\" (UniqueName: \"kubernetes.io/projected/c44f1e05-226d-40c9-8626-484e90514cbb-kube-api-access-ptghk\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.931362 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.931241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c44f1e05-226d-40c9-8626-484e90514cbb-manager-config\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.931362 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.931273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c44f1e05-226d-40c9-8626-484e90514cbb-metrics-cert\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.931362 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.931292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c44f1e05-226d-40c9-8626-484e90514cbb-cert\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.931933 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.931903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c44f1e05-226d-40c9-8626-484e90514cbb-manager-config\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.933733 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.933710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c44f1e05-226d-40c9-8626-484e90514cbb-cert\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.933834 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.933749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c44f1e05-226d-40c9-8626-484e90514cbb-metrics-cert\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:00.954466 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:00.954441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptghk\" (UniqueName: \"kubernetes.io/projected/c44f1e05-226d-40c9-8626-484e90514cbb-kube-api-access-ptghk\") pod \"lws-controller-manager-9fc458d94-ds2ph\" (UID: \"c44f1e05-226d-40c9-8626-484e90514cbb\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:01.067128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:01.067110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:01.263712 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:01.263671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:01.357477 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:01.357444 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph"] Sep 10 17:18:01.358596 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:18:01.358573 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44f1e05_226d_40c9_8626_484e90514cbb.slice/crio-183842f488aa3e1a12541f001c72d20a74551b479cddb1a2e784881caaf0a16c WatchSource:0}: Error finding container 183842f488aa3e1a12541f001c72d20a74551b479cddb1a2e784881caaf0a16c: Status 404 returned error can't find the container with id 183842f488aa3e1a12541f001c72d20a74551b479cddb1a2e784881caaf0a16c Sep 10 17:18:01.419340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:01.419309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:01.618409 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:01.618316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:02.023407 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:02.023318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:02.181803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:02.181771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" event={"ID":"c44f1e05-226d-40c9-8626-484e90514cbb","Type":"ContainerStarted","Data":"183842f488aa3e1a12541f001c72d20a74551b479cddb1a2e784881caaf0a16c"} Sep 10 17:18:03.193515 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:03.193473 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" event={"ID":"c44f1e05-226d-40c9-8626-484e90514cbb","Type":"ContainerStarted","Data":"6448bee98c62687fd96a08695111d38b7952a4b2c12ac2f244503e17c4f4f8a8"} Sep 10 17:18:03.193981 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:03.193597 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:03.220717 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:03.220666 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" podStartSLOduration=1.508752227 podStartE2EDuration="3.220651239s" totalImagesPullingTime="1.711899012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:03.218379894 +0000 UTC m=+389.199459892" watchObservedRunningTime="2026-09-10 17:18:03.220651239 +0000 UTC m=+389.201731237" Sep 10 17:18:03.583693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:03.583665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:03.695865 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:03.695831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:05.081172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:05.081135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:09.014410 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.014372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:09.160499 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.160467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:09.261630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.261594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:09.445102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.445069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz"] Sep 10 17:18:09.450049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.450031 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.452766 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.452749 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 17:18:09.452933 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.452915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7wn5c\"" Sep 10 17:18:09.461189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.461170 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz"] Sep 10 17:18:09.481696 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.481666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:09.604254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.604218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:09.606242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292fd1a-ada8-458b-8486-e24743c1ccb2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmxfj\" (UniqueName: \"kubernetes.io/projected/6292fd1a-ada8-458b-8486-e24743c1ccb2-kube-api-access-kmxfj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606584 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.606584 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.606511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.707840 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.707780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.707840 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.707824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmxfj\" (UniqueName: \"kubernetes.io/projected/6292fd1a-ada8-458b-8486-e24743c1ccb2-kube-api-access-kmxfj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.707860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.707931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.707955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.707974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.707988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.708005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292fd1a-ada8-458b-8486-e24743c1ccb2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.708040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708411 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.708383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708577 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.708478 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708577 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.708512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.708750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.708909 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.708869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292fd1a-ada8-458b-8486-e24743c1ccb2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.710242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.710216 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.710475 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.710456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.717714 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.717692 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292fd1a-ada8-458b-8486-e24743c1ccb2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.718351 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.718328 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmxfj\" (UniqueName: \"kubernetes.io/projected/6292fd1a-ada8-458b-8486-e24743c1ccb2-kube-api-access-kmxfj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz\" (UID: \"6292fd1a-ada8-458b-8486-e24743c1ccb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.759962 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.759933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:09.840525 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:09.840491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:10.035615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:10.035592 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz"] Sep 10 17:18:10.039016 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:18:10.038984 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6292fd1a_ada8_458b_8486_e24743c1ccb2.slice/crio-385da6881e9f62f756ea80dbe7c263ad698c25d03a0a2573fe7b1ab810323203 WatchSource:0}: Error finding container 385da6881e9f62f756ea80dbe7c263ad698c25d03a0a2573fe7b1ab810323203: Status 404 returned error can't find the container with id 385da6881e9f62f756ea80dbe7c263ad698c25d03a0a2573fe7b1ab810323203 Sep 10 17:18:10.225404 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:10.225372 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" event={"ID":"6292fd1a-ada8-458b-8486-e24743c1ccb2","Type":"ContainerStarted","Data":"385da6881e9f62f756ea80dbe7c263ad698c25d03a0a2573fe7b1ab810323203"} Sep 10 17:18:12.327734 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.327699 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wgc26"] Sep 10 17:18:12.334805 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.334781 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.337843 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.337807 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 17:18:12.337996 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.337949 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xnfkz\"" Sep 10 17:18:12.349262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.349232 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wgc26"] Sep 10 17:18:12.372937 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.372907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:12.431295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.431267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6x2\" (UniqueName: \"kubernetes.io/projected/efbf1d1e-8071-4771-85a4-a283330d3af0-kube-api-access-hg6x2\") pod \"odh-model-controller-7998c67dc4-wgc26\" (UID: \"efbf1d1e-8071-4771-85a4-a283330d3af0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.431441 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.431384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efbf1d1e-8071-4771-85a4-a283330d3af0-cert\") pod \"odh-model-controller-7998c67dc4-wgc26\" (UID: \"efbf1d1e-8071-4771-85a4-a283330d3af0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.467629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.467592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:12.532593 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.532561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6x2\" (UniqueName: \"kubernetes.io/projected/efbf1d1e-8071-4771-85a4-a283330d3af0-kube-api-access-hg6x2\") pod \"odh-model-controller-7998c67dc4-wgc26\" (UID: \"efbf1d1e-8071-4771-85a4-a283330d3af0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.532749 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.532646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efbf1d1e-8071-4771-85a4-a283330d3af0-cert\") pod \"odh-model-controller-7998c67dc4-wgc26\" (UID: \"efbf1d1e-8071-4771-85a4-a283330d3af0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.535111 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.535079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efbf1d1e-8071-4771-85a4-a283330d3af0-cert\") pod \"odh-model-controller-7998c67dc4-wgc26\" (UID: \"efbf1d1e-8071-4771-85a4-a283330d3af0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.544030 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.544009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6x2\" (UniqueName: \"kubernetes.io/projected/efbf1d1e-8071-4771-85a4-a283330d3af0-kube-api-access-hg6x2\") pod \"odh-model-controller-7998c67dc4-wgc26\" (UID: \"efbf1d1e-8071-4771-85a4-a283330d3af0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.571084 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.571041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:12.629681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.629647 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:18:12.629753 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.629717 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:18:12.629753 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.629745 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:18:12.647010 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.646984 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:12.898182 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.898098 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tn44v"] Sep 10 17:18:12.901442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.901419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:12.904182 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.904160 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 17:18:12.911725 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.911702 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tn44v"] Sep 10 17:18:12.924488 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.924466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:12.929033 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:12.929014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wgc26"] Sep 10 17:18:12.932385 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:18:12.932323 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbf1d1e_8071_4771_85a4_a283330d3af0.slice/crio-17a9fa3366982980a0b8e8469fe9cad961360a35438f5125b78bf2ee9df28c20 WatchSource:0}: Error finding container 17a9fa3366982980a0b8e8469fe9cad961360a35438f5125b78bf2ee9df28c20: Status 404 returned error can't find the container with id 17a9fa3366982980a0b8e8469fe9cad961360a35438f5125b78bf2ee9df28c20 Sep 10 17:18:13.038425 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.038375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvkq\" (UniqueName: \"kubernetes.io/projected/f1a83d70-1e1f-4b39-b07a-402432beb414-kube-api-access-qlvkq\") pod \"model-serving-api-54d89bd5fc-tn44v\" (UID: \"f1a83d70-1e1f-4b39-b07a-402432beb414\") " pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:13.038574 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.038475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a83d70-1e1f-4b39-b07a-402432beb414-tls-certs\") pod \"model-serving-api-54d89bd5fc-tn44v\" (UID: \"f1a83d70-1e1f-4b39-b07a-402432beb414\") " pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:13.139745 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.139712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvkq\" (UniqueName: \"kubernetes.io/projected/f1a83d70-1e1f-4b39-b07a-402432beb414-kube-api-access-qlvkq\") pod \"model-serving-api-54d89bd5fc-tn44v\" (UID: \"f1a83d70-1e1f-4b39-b07a-402432beb414\") " pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:13.139864 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.139759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a83d70-1e1f-4b39-b07a-402432beb414-tls-certs\") pod \"model-serving-api-54d89bd5fc-tn44v\" (UID: \"f1a83d70-1e1f-4b39-b07a-402432beb414\") " pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:13.139943 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:18:13.139916 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 17:18:13.140014 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:18:13.140004 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1a83d70-1e1f-4b39-b07a-402432beb414-tls-certs podName:f1a83d70-1e1f-4b39-b07a-402432beb414 nodeName:}" failed. No retries permitted until 2026-09-10 17:18:13.639983574 +0000 UTC m=+399.621063562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f1a83d70-1e1f-4b39-b07a-402432beb414-tls-certs") pod "model-serving-api-54d89bd5fc-tn44v" (UID: "f1a83d70-1e1f-4b39-b07a-402432beb414") : secret "model-serving-api-tls" not found Sep 10 17:18:13.149542 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.149482 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvkq\" (UniqueName: \"kubernetes.io/projected/f1a83d70-1e1f-4b39-b07a-402432beb414-kube-api-access-qlvkq\") pod \"model-serving-api-54d89bd5fc-tn44v\" (UID: \"f1a83d70-1e1f-4b39-b07a-402432beb414\") " pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:13.241747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.241712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" event={"ID":"efbf1d1e-8071-4771-85a4-a283330d3af0","Type":"ContainerStarted","Data":"17a9fa3366982980a0b8e8469fe9cad961360a35438f5125b78bf2ee9df28c20"} Sep 10 17:18:13.243176 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.243151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" event={"ID":"6292fd1a-ada8-458b-8486-e24743c1ccb2","Type":"ContainerStarted","Data":"a460d6d419fd713b8d2cf5aca92cab4a6dc852cb767ac80aa4657e6767c91b48"} Sep 10 17:18:13.266068 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.266026 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" podStartSLOduration=1.6775715519999999 podStartE2EDuration="4.266011676s" totalImagesPullingTime="2.588440124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:13.265063559 +0000 UTC m=+399.246143558" watchObservedRunningTime="2026-09-10 17:18:13.266011676 +0000 UTC m=+399.247091675" Sep 10 17:18:13.520407 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.520324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:13.643933 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.643901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a83d70-1e1f-4b39-b07a-402432beb414-tls-certs\") pod \"model-serving-api-54d89bd5fc-tn44v\" (UID: \"f1a83d70-1e1f-4b39-b07a-402432beb414\") " pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:13.646965 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.646915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a83d70-1e1f-4b39-b07a-402432beb414-tls-certs\") pod \"model-serving-api-54d89bd5fc-tn44v\" (UID: \"f1a83d70-1e1f-4b39-b07a-402432beb414\") " pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:13.760636 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.760603 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:13.766311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.766279 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:13.813875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:13.813769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:14.104312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:14.104287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tn44v"] Sep 10 17:18:14.106655 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:18:14.106623 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a83d70_1e1f_4b39_b07a_402432beb414.slice/crio-7454dfcedc84ef778171c7b4149f2ed8e92950746d9c80ee1878fa646447c773 WatchSource:0}: Error finding container 7454dfcedc84ef778171c7b4149f2ed8e92950746d9c80ee1878fa646447c773: Status 404 returned error can't find the container with id 7454dfcedc84ef778171c7b4149f2ed8e92950746d9c80ee1878fa646447c773 Sep 10 17:18:14.201005 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:14.200977 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-ds2ph" Sep 10 17:18:14.249334 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:14.249299 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" event={"ID":"f1a83d70-1e1f-4b39-b07a-402432beb414","Type":"ContainerStarted","Data":"7454dfcedc84ef778171c7b4149f2ed8e92950746d9c80ee1878fa646447c773"} Sep 10 17:18:14.249540 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:14.249519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:14.250573 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:14.250553 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz" Sep 10 17:18:16.262687 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:16.262601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" event={"ID":"efbf1d1e-8071-4771-85a4-a283330d3af0","Type":"ContainerStarted","Data":"0f917a45a4046536d01a06434e2532f92c6731cc74dd2d82b498813300e6e512"} Sep 10 17:18:16.263147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:16.262863 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:16.281914 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:16.281845 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" podStartSLOduration=1.296042789 podStartE2EDuration="4.281823335s" totalImagesPullingTime="2.985780546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:16.27895017 +0000 UTC m=+402.260030169" watchObservedRunningTime="2026-09-10 17:18:16.281823335 +0000 UTC m=+402.262903317" Sep 10 17:18:18.274469 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:18.274429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" event={"ID":"f1a83d70-1e1f-4b39-b07a-402432beb414","Type":"ContainerStarted","Data":"a0e59220c25193f92d8fbf83ba951c6345011f965edc25ed6dc0606461c46d1c"} Sep 10 17:18:18.274987 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:18.274580 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:18.314013 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:18.313965 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" podStartSLOduration=2.778828934 podStartE2EDuration="6.313950462s" totalImagesPullingTime="3.535121528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:18.312129638 +0000 UTC m=+404.293209628" watchObservedRunningTime="2026-09-10 17:18:18.313950462 +0000 UTC m=+404.295030460" Sep 10 17:18:22.269372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.269335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:22.361571 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.361539 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw"] Sep 10 17:18:22.364451 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.364434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:22.367153 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.367131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9vxp2\"" Sep 10 17:18:22.367392 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.367374 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 17:18:22.377007 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.376982 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw"] Sep 10 17:18:22.419946 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.419915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:22.520172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.520090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-cert\") pod \"kserve-controller-manager-d6f4b65b8-lnsmw\" (UID: \"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:22.520172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.520144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wlq2\" (UniqueName: \"kubernetes.io/projected/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-kube-api-access-6wlq2\") pod \"kserve-controller-manager-d6f4b65b8-lnsmw\" (UID: \"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:22.620938 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.620874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wlq2\" (UniqueName: \"kubernetes.io/projected/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-kube-api-access-6wlq2\") pod \"kserve-controller-manager-d6f4b65b8-lnsmw\" (UID: \"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:22.621090 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.621008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-cert\") pod \"kserve-controller-manager-d6f4b65b8-lnsmw\" (UID: \"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:22.621132 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:18:22.621115 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 10 17:18:22.621188 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:18:22.621177 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-cert podName:98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17 nodeName:}" failed. No retries permitted until 2026-09-10 17:18:23.121157061 +0000 UTC m=+409.102237053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-cert") pod "kserve-controller-manager-d6f4b65b8-lnsmw" (UID: "98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17") : secret "kserve-webhook-server-cert" not found Sep 10 17:18:22.659398 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.659366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wlq2\" (UniqueName: \"kubernetes.io/projected/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-kube-api-access-6wlq2\") pod \"kserve-controller-manager-d6f4b65b8-lnsmw\" (UID: \"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:22.744799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.744760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:22.924259 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:22.924226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:23.124482 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:23.124441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-cert\") pod \"kserve-controller-manager-d6f4b65b8-lnsmw\" (UID: \"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:23.126759 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:23.126731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17-cert\") pod \"kserve-controller-manager-d6f4b65b8-lnsmw\" (UID: \"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:23.274276 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:23.274202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:23.554456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:23.554431 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw"] Sep 10 17:18:23.556008 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:18:23.555980 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cbdd4c_bcfa_4ef7_85a4_9e2c40f58a17.slice/crio-79b309c97ab19404fd569d185582a20c734c4d4ab8e156e0c1fd57f38ae8940c WatchSource:0}: Error finding container 79b309c97ab19404fd569d185582a20c734c4d4ab8e156e0c1fd57f38ae8940c: Status 404 returned error can't find the container with id 79b309c97ab19404fd569d185582a20c734c4d4ab8e156e0c1fd57f38ae8940c Sep 10 17:18:23.605346 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:23.605314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:23.714734 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:23.714702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:23.827387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:23.827353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:24.305946 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:24.305855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" event={"ID":"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17","Type":"ContainerStarted","Data":"79b309c97ab19404fd569d185582a20c734c4d4ab8e156e0c1fd57f38ae8940c"} Sep 10 17:18:26.316260 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:26.316232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" event={"ID":"98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17","Type":"ContainerStarted","Data":"05dee9a345117fbfe26b98c9e7292c5f52e1f2ebe8522ffe797435f594185260"} Sep 10 17:18:26.316596 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:26.316375 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:18:26.356258 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:26.356209 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" podStartSLOduration=1.856453954 podStartE2EDuration="4.356192285s" totalImagesPullingTime="2.499738331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:26.354002154 +0000 UTC m=+412.335082156" watchObservedRunningTime="2026-09-10 17:18:26.356192285 +0000 UTC m=+412.337272284" Sep 10 17:18:27.287383 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:27.287348 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wgc26" Sep 10 17:18:29.283631 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:29.283602 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tn44v" Sep 10 17:18:32.740278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.740238 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56bf68d87d-zdpfh"] Sep 10 17:18:32.743829 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.743805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.761407 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.761384 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bf68d87d-zdpfh"] Sep 10 17:18:32.820606 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.820572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:32.909846 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.909804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-console-config\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.909846 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.909849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hds4b\" (UniqueName: \"kubernetes.io/projected/ac274796-d611-46d9-8781-1841abfb12d8-kube-api-access-hds4b\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.910114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.909982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac274796-d611-46d9-8781-1841abfb12d8-console-oauth-config\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.910114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.910019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-oauth-serving-cert\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.910114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.910082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-trusted-ca-bundle\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.910253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.910114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-service-ca\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.910253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.910146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac274796-d611-46d9-8781-1841abfb12d8-console-serving-cert\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:32.920680 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.920641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:32.992786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:32.992703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:33.010738 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.010704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-console-config\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.010738 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.010745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hds4b\" (UniqueName: \"kubernetes.io/projected/ac274796-d611-46d9-8781-1841abfb12d8-kube-api-access-hds4b\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.010861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.010785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac274796-d611-46d9-8781-1841abfb12d8-console-oauth-config\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.010861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.010803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-oauth-serving-cert\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.010861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.010829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-trusted-ca-bundle\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.010861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.010848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-service-ca\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.011087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.010867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac274796-d611-46d9-8781-1841abfb12d8-console-serving-cert\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.011594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.011563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-console-config\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.011682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.011563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-oauth-serving-cert\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.011682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.011567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-service-ca\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.011824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.011801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac274796-d611-46d9-8781-1841abfb12d8-trusted-ca-bundle\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.013354 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.013336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac274796-d611-46d9-8781-1841abfb12d8-console-oauth-config\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.013528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.013412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac274796-d611-46d9-8781-1841abfb12d8-console-serving-cert\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.019961 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.019940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hds4b\" (UniqueName: \"kubernetes.io/projected/ac274796-d611-46d9-8781-1841abfb12d8-kube-api-access-hds4b\") pod \"console-56bf68d87d-zdpfh\" (UID: \"ac274796-d611-46d9-8781-1841abfb12d8\") " pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.052805 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.052781 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:33.330207 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.330177 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bf68d87d-zdpfh"] Sep 10 17:18:33.333389 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:18:33.333366 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac274796_d611_46d9_8781_1841abfb12d8.slice/crio-3d9bf18bc6b84d904130744d5ee43961aaf6becd2c47225c3cda6ee4dec94235 WatchSource:0}: Error finding container 3d9bf18bc6b84d904130744d5ee43961aaf6becd2c47225c3cda6ee4dec94235: Status 404 returned error can't find the container with id 3d9bf18bc6b84d904130744d5ee43961aaf6becd2c47225c3cda6ee4dec94235 Sep 10 17:18:33.363018 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.362990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bf68d87d-zdpfh" event={"ID":"ac274796-d611-46d9-8781-1841abfb12d8","Type":"ContainerStarted","Data":"3d9bf18bc6b84d904130744d5ee43961aaf6becd2c47225c3cda6ee4dec94235"} Sep 10 17:18:33.707014 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.706922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:33.774220 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:33.774177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:34.369349 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:34.369313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bf68d87d-zdpfh" event={"ID":"ac274796-d611-46d9-8781-1841abfb12d8","Type":"ContainerStarted","Data":"d643e721f41242764082c92de1c2a90ccf179aaf2168cdbc78a0d5d6947c8fd9"} Sep 10 17:18:34.391615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:34.391570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56bf68d87d-zdpfh" podStartSLOduration=2.391556814 podStartE2EDuration="2.391556814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:34.388910151 +0000 UTC m=+420.369990140" watchObservedRunningTime="2026-09-10 17:18:34.391556814 +0000 UTC m=+420.372636870" Sep 10 17:18:43.053467 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:43.053433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:43.054007 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:43.053500 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:43.058047 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:43.058021 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:43.413081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:43.413053 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56bf68d87d-zdpfh" Sep 10 17:18:43.466640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:43.466598 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c47bb4df-j78rl"] Sep 10 17:18:43.635806 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:43.635768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:43.735516 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:43.735429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:18:57.328295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:18:57.328211 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lnsmw" Sep 10 17:19:04.410461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:04.410427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:04.518380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:04.518346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:08.487922 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.487870 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64c47bb4df-j78rl" podUID="ea694fba-f496-4702-844f-5eeabda43f3a" containerName="console" containerID="cri-o://04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa" gracePeriod=15 Sep 10 17:19:08.723768 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.723745 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:19:08.801363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801288 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcgqn\" (UniqueName: \"kubernetes.io/projected/ea694fba-f496-4702-844f-5eeabda43f3a-kube-api-access-qcgqn\") pod \"ea694fba-f496-4702-844f-5eeabda43f3a\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " Sep 10 17:19:08.801363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801335 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-trusted-ca-bundle\") pod \"ea694fba-f496-4702-844f-5eeabda43f3a\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " Sep 10 17:19:08.801557 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801405 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-console-config\") pod \"ea694fba-f496-4702-844f-5eeabda43f3a\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " Sep 10 17:19:08.801557 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-service-ca\") pod \"ea694fba-f496-4702-844f-5eeabda43f3a\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " Sep 10 17:19:08.801557 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801487 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-oauth-serving-cert\") pod \"ea694fba-f496-4702-844f-5eeabda43f3a\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " Sep 10 17:19:08.801557 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801538 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-serving-cert\") pod \"ea694fba-f496-4702-844f-5eeabda43f3a\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " Sep 10 17:19:08.801751 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-oauth-config\") pod \"ea694fba-f496-4702-844f-5eeabda43f3a\" (UID: \"ea694fba-f496-4702-844f-5eeabda43f3a\") " Sep 10 17:19:08.801799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801771 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ea694fba-f496-4702-844f-5eeabda43f3a" (UID: "ea694fba-f496-4702-844f-5eeabda43f3a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:19:08.801897 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801816 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-service-ca" (OuterVolumeSpecName: "service-ca") pod "ea694fba-f496-4702-844f-5eeabda43f3a" (UID: "ea694fba-f496-4702-844f-5eeabda43f3a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:19:08.801897 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.801831 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-console-config" (OuterVolumeSpecName: "console-config") pod "ea694fba-f496-4702-844f-5eeabda43f3a" (UID: "ea694fba-f496-4702-844f-5eeabda43f3a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:19:08.804189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.802303 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ea694fba-f496-4702-844f-5eeabda43f3a" (UID: "ea694fba-f496-4702-844f-5eeabda43f3a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:19:08.805936 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.804360 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea694fba-f496-4702-844f-5eeabda43f3a-kube-api-access-qcgqn" (OuterVolumeSpecName: "kube-api-access-qcgqn") pod "ea694fba-f496-4702-844f-5eeabda43f3a" (UID: "ea694fba-f496-4702-844f-5eeabda43f3a"). InnerVolumeSpecName "kube-api-access-qcgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:19:08.805936 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.804568 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ea694fba-f496-4702-844f-5eeabda43f3a" (UID: "ea694fba-f496-4702-844f-5eeabda43f3a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:19:08.805936 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.804656 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ea694fba-f496-4702-844f-5eeabda43f3a" (UID: "ea694fba-f496-4702-844f-5eeabda43f3a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:19:08.902991 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.902964 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcgqn\" (UniqueName: \"kubernetes.io/projected/ea694fba-f496-4702-844f-5eeabda43f3a-kube-api-access-qcgqn\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:08.902991 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.902986 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-trusted-ca-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:08.903139 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.902996 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-console-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:08.903139 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.903005 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:08.903139 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.903013 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea694fba-f496-4702-844f-5eeabda43f3a-oauth-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:08.903139 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.903021 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-serving-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:08.903139 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:08.903028 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea694fba-f496-4702-844f-5eeabda43f3a-console-oauth-config\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:09.528310 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.528273 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea694fba-f496-4702-844f-5eeabda43f3a" containerID="04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa" exitCode=0 Sep 10 17:19:09.528797 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.528333 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c47bb4df-j78rl" Sep 10 17:19:09.528797 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.528351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c47bb4df-j78rl" event={"ID":"ea694fba-f496-4702-844f-5eeabda43f3a","Type":"ContainerDied","Data":"04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa"} Sep 10 17:19:09.528797 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.528398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c47bb4df-j78rl" event={"ID":"ea694fba-f496-4702-844f-5eeabda43f3a","Type":"ContainerDied","Data":"52e5044248225aff634154cf72a4e67f37fca61d2e02f50bc2814e94f5f3a6a1"} Sep 10 17:19:09.528797 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.528417 2576 scope.go:117] "RemoveContainer" containerID="04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa" Sep 10 17:19:09.537448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.537431 2576 scope.go:117] "RemoveContainer" containerID="04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa" Sep 10 17:19:09.537690 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:19:09.537673 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa\": container with ID starting with 04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa not found: ID does not exist" containerID="04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa" Sep 10 17:19:09.537737 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.537698 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa"} err="failed to get container status \"04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa\": rpc error: code = NotFound desc = could not find container \"04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa\": container with ID starting with 04de5ee3edfbc29f17e0416b82237e28f55b8b31b379e92c715c182d11e1ebaa not found: ID does not exist" Sep 10 17:19:09.551539 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.551514 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c47bb4df-j78rl"] Sep 10 17:19:09.559754 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:09.559733 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64c47bb4df-j78rl"] Sep 10 17:19:10.485696 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:10.485666 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea694fba-f496-4702-844f-5eeabda43f3a" path="/var/lib/kubelet/pods/ea694fba-f496-4702-844f-5eeabda43f3a/volumes" Sep 10 17:19:13.262068 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:13.262034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:13.354255 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:13.354203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:18.487791 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.487760 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85"] Sep 10 17:19:18.488174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.488088 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea694fba-f496-4702-844f-5eeabda43f3a" containerName="console" Sep 10 17:19:18.488174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.488100 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea694fba-f496-4702-844f-5eeabda43f3a" containerName="console" Sep 10 17:19:18.488249 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.488182 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea694fba-f496-4702-844f-5eeabda43f3a" containerName="console" Sep 10 17:19:18.492633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.492613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.498071 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.498050 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 17:19:18.498192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.498055 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 17:19:18.499194 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.499175 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-96ksv\"" Sep 10 17:19:18.505541 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.505516 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85"] Sep 10 17:19:18.581431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.581398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.581630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.581439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95ql\" (UniqueName: \"kubernetes.io/projected/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-kube-api-access-w95ql\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.581630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.581536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.585368 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.585344 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v"] Sep 10 17:19:18.588904 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.588888 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.598384 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.598357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v"] Sep 10 17:19:18.682594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.682559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt7kk\" (UniqueName: \"kubernetes.io/projected/31ca1558-f25d-4f34-927d-0610e34b11d8-kube-api-access-bt7kk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.682779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.682610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.682779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.682633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.682779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.682681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95ql\" (UniqueName: \"kubernetes.io/projected/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-kube-api-access-w95ql\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.682779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.682707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.682779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.682738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.683069 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.683052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.683107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.683054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.683466 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.683448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk"] Sep 10 17:19:18.686913 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.686898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.693376 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.693354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95ql\" (UniqueName: \"kubernetes.io/projected/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-kube-api-access-w95ql\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.698210 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.698186 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk"] Sep 10 17:19:18.783725 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.783645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.783725 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.783691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flt9q\" (UniqueName: \"kubernetes.io/projected/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-kube-api-access-flt9q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.783936 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.783725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.783936 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.783761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7kk\" (UniqueName: \"kubernetes.io/projected/31ca1558-f25d-4f34-927d-0610e34b11d8-kube-api-access-bt7kk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.783936 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.783785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.783936 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.783810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.784122 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.784102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.784160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.784129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.792836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.792812 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf"] Sep 10 17:19:18.796358 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.796340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.797869 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.797851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7kk\" (UniqueName: \"kubernetes.io/projected/31ca1558-f25d-4f34-927d-0610e34b11d8-kube-api-access-bt7kk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.801650 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.801636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:18.812422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.808542 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf"] Sep 10 17:19:18.884759 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.884696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.884759 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.884740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmvs\" (UniqueName: \"kubernetes.io/projected/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-kube-api-access-bfmvs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.884953 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.884780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flt9q\" (UniqueName: \"kubernetes.io/projected/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-kube-api-access-flt9q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.884953 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.884853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.884953 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.884927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.885051 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.885011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.885332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.885315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.885425 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.885405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.893627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.893593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flt9q\" (UniqueName: \"kubernetes.io/projected/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-kube-api-access-flt9q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:18.898256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.898234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:18.985655 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.985628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.985830 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.985718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.985830 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.985761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmvs\" (UniqueName: \"kubernetes.io/projected/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-kube-api-access-bfmvs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.986058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.986032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.986147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.986045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:18.994867 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:18.994839 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmvs\" (UniqueName: \"kubernetes.io/projected/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-kube-api-access-bfmvs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:19.003627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.003610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:19.083966 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.083943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85"] Sep 10 17:19:19.085029 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:19:19.085006 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9b248e_c4c1_4f78_99e0_eda1cc28dde4.slice/crio-542e761d61f188d61bd13118fe0eee4734f84a0e1b03be8f23ac83ef83975ad0 WatchSource:0}: Error finding container 542e761d61f188d61bd13118fe0eee4734f84a0e1b03be8f23ac83ef83975ad0: Status 404 returned error can't find the container with id 542e761d61f188d61bd13118fe0eee4734f84a0e1b03be8f23ac83ef83975ad0 Sep 10 17:19:19.131637 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.131612 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:19.174514 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.174482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v"] Sep 10 17:19:19.176070 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:19:19.176045 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca1558_f25d_4f34_927d_0610e34b11d8.slice/crio-66e34b6baba8b7e426207ef186bf96dfb48aa2573d18c706c2c2e18bc138bc42 WatchSource:0}: Error finding container 66e34b6baba8b7e426207ef186bf96dfb48aa2573d18c706c2c2e18bc138bc42: Status 404 returned error can't find the container with id 66e34b6baba8b7e426207ef186bf96dfb48aa2573d18c706c2c2e18bc138bc42 Sep 10 17:19:19.275446 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.275421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk"] Sep 10 17:19:19.276302 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:19:19.276278 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3f8071_4c37_4d8f_94f3_0a1b68d67c50.slice/crio-30688dd83b53d86897d1a692b163ae44b318cccc8a9e71f0af8cf21c4217c3c1 WatchSource:0}: Error finding container 30688dd83b53d86897d1a692b163ae44b318cccc8a9e71f0af8cf21c4217c3c1: Status 404 returned error can't find the container with id 30688dd83b53d86897d1a692b163ae44b318cccc8a9e71f0af8cf21c4217c3c1 Sep 10 17:19:19.411921 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.411874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf"] Sep 10 17:19:19.429696 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:19:19.429670 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0e0195_45b4_408f_a189_fd00ab3f0bb4.slice/crio-0610ccf36b17bf90c2270b655e3f64cc747a651e0334f1a8f7d01733b9e16d71 WatchSource:0}: Error finding container 0610ccf36b17bf90c2270b655e3f64cc747a651e0334f1a8f7d01733b9e16d71: Status 404 returned error can't find the container with id 0610ccf36b17bf90c2270b655e3f64cc747a651e0334f1a8f7d01733b9e16d71 Sep 10 17:19:19.575041 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.575012 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerID="beef5e97689330b3a04899383346929671c8fc778fcba925e8df88e4cd3af039" exitCode=0 Sep 10 17:19:19.575418 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.575090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" event={"ID":"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4","Type":"ContainerDied","Data":"beef5e97689330b3a04899383346929671c8fc778fcba925e8df88e4cd3af039"} Sep 10 17:19:19.575418 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.575148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" event={"ID":"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4","Type":"ContainerStarted","Data":"542e761d61f188d61bd13118fe0eee4734f84a0e1b03be8f23ac83ef83975ad0"} Sep 10 17:19:19.576690 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.576664 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerID="6c0e5cab37cabda11c47ab0ff8b09ff89ea4e463c09bb3c82bdf201708685587" exitCode=0 Sep 10 17:19:19.576805 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.576726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" event={"ID":"5e0e0195-45b4-408f-a189-fd00ab3f0bb4","Type":"ContainerDied","Data":"6c0e5cab37cabda11c47ab0ff8b09ff89ea4e463c09bb3c82bdf201708685587"} Sep 10 17:19:19.576805 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.576774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" event={"ID":"5e0e0195-45b4-408f-a189-fd00ab3f0bb4","Type":"ContainerStarted","Data":"0610ccf36b17bf90c2270b655e3f64cc747a651e0334f1a8f7d01733b9e16d71"} Sep 10 17:19:19.578568 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.578547 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerID="9bf37368eb96b3725131eec58f1fa0c108be24e372d21d48ef87d60fdcfc3664" exitCode=0 Sep 10 17:19:19.578667 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.578635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" event={"ID":"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50","Type":"ContainerDied","Data":"9bf37368eb96b3725131eec58f1fa0c108be24e372d21d48ef87d60fdcfc3664"} Sep 10 17:19:19.578795 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.578678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" event={"ID":"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50","Type":"ContainerStarted","Data":"30688dd83b53d86897d1a692b163ae44b318cccc8a9e71f0af8cf21c4217c3c1"} Sep 10 17:19:19.580493 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.580456 2576 generic.go:356] "Generic (PLEG): container finished" podID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerID="227195d490c74c9bb2e68d4f5ae278035000facc07f90d331a9d7bbe90cf3c10" exitCode=0 Sep 10 17:19:19.580493 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.580482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" event={"ID":"31ca1558-f25d-4f34-927d-0610e34b11d8","Type":"ContainerDied","Data":"227195d490c74c9bb2e68d4f5ae278035000facc07f90d331a9d7bbe90cf3c10"} Sep 10 17:19:19.580602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:19.580523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" event={"ID":"31ca1558-f25d-4f34-927d-0610e34b11d8","Type":"ContainerStarted","Data":"66e34b6baba8b7e426207ef186bf96dfb48aa2573d18c706c2c2e18bc138bc42"} Sep 10 17:19:20.587345 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:20.587314 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerID="6e936f7613a6aaede07f6e0359905e6997d72a56c5e8a73b854810c057cb5d11" exitCode=0 Sep 10 17:19:20.587687 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:20.587353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" event={"ID":"5e0e0195-45b4-408f-a189-fd00ab3f0bb4","Type":"ContainerDied","Data":"6e936f7613a6aaede07f6e0359905e6997d72a56c5e8a73b854810c057cb5d11"} Sep 10 17:19:21.593649 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.593616 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerID="e2f9a89fc52bfd167c98adc00ebc8587037ad5f009995fdbb7cce98a814ad721" exitCode=0 Sep 10 17:19:21.594098 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.593701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" event={"ID":"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50","Type":"ContainerDied","Data":"e2f9a89fc52bfd167c98adc00ebc8587037ad5f009995fdbb7cce98a814ad721"} Sep 10 17:19:21.595797 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.595773 2576 generic.go:356] "Generic (PLEG): container finished" podID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerID="726975304fd7d600f6da8fa5e8a91f9f29f77fe776b378f356e4039bd69aeac8" exitCode=0 Sep 10 17:19:21.595903 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.595852 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" event={"ID":"31ca1558-f25d-4f34-927d-0610e34b11d8","Type":"ContainerDied","Data":"726975304fd7d600f6da8fa5e8a91f9f29f77fe776b378f356e4039bd69aeac8"} Sep 10 17:19:21.597793 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.597769 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerID="a3df1c00962a05b16bc03383430e058a52a0a0665e54f9226efca3336e0abf88" exitCode=0 Sep 10 17:19:21.597896 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.597850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" event={"ID":"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4","Type":"ContainerDied","Data":"a3df1c00962a05b16bc03383430e058a52a0a0665e54f9226efca3336e0abf88"} Sep 10 17:19:21.600163 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.600142 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerID="3d0ed25b6a1111daf56275e554bdaa609d562aedd810676fe8fd3ecabe02bfd9" exitCode=0 Sep 10 17:19:21.600253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:21.600173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" event={"ID":"5e0e0195-45b4-408f-a189-fd00ab3f0bb4","Type":"ContainerDied","Data":"3d0ed25b6a1111daf56275e554bdaa609d562aedd810676fe8fd3ecabe02bfd9"} Sep 10 17:19:22.607204 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.607169 2576 generic.go:356] "Generic (PLEG): container finished" podID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerID="c68449a3902eb731d513471d129d3a4a18dba38c46ec668911e9bb94909e30af" exitCode=0 Sep 10 17:19:22.607626 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.607260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" event={"ID":"31ca1558-f25d-4f34-927d-0610e34b11d8","Type":"ContainerDied","Data":"c68449a3902eb731d513471d129d3a4a18dba38c46ec668911e9bb94909e30af"} Sep 10 17:19:22.609300 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.609276 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerID="1028f3972ea0b8570a98d344408df70d8cb9005c11d7b6eae5fba1095a3ba8cd" exitCode=0 Sep 10 17:19:22.609424 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.609359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" event={"ID":"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4","Type":"ContainerDied","Data":"1028f3972ea0b8570a98d344408df70d8cb9005c11d7b6eae5fba1095a3ba8cd"} Sep 10 17:19:22.611538 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.611517 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerID="04b8aebb1dfd4660b8447de0dc53eff6f2d3ab7091505dc643b7bbbd80aff015" exitCode=0 Sep 10 17:19:22.611637 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.611535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" event={"ID":"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50","Type":"ContainerDied","Data":"04b8aebb1dfd4660b8447de0dc53eff6f2d3ab7091505dc643b7bbbd80aff015"} Sep 10 17:19:22.744099 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.744075 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:22.818292 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.818260 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-bundle\") pod \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " Sep 10 17:19:22.818469 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.818324 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfmvs\" (UniqueName: \"kubernetes.io/projected/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-kube-api-access-bfmvs\") pod \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " Sep 10 17:19:22.818469 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.818378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-util\") pod \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\" (UID: \"5e0e0195-45b4-408f-a189-fd00ab3f0bb4\") " Sep 10 17:19:22.818866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.818829 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-bundle" (OuterVolumeSpecName: "bundle") pod "5e0e0195-45b4-408f-a189-fd00ab3f0bb4" (UID: "5e0e0195-45b4-408f-a189-fd00ab3f0bb4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:22.820386 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.820365 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-kube-api-access-bfmvs" (OuterVolumeSpecName: "kube-api-access-bfmvs") pod "5e0e0195-45b4-408f-a189-fd00ab3f0bb4" (UID: "5e0e0195-45b4-408f-a189-fd00ab3f0bb4"). InnerVolumeSpecName "kube-api-access-bfmvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:19:22.825264 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.825214 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-util" (OuterVolumeSpecName: "util") pod "5e0e0195-45b4-408f-a189-fd00ab3f0bb4" (UID: "5e0e0195-45b4-408f-a189-fd00ab3f0bb4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:22.919311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.919225 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:22.919311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.919256 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:22.919311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:22.919270 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmvs\" (UniqueName: \"kubernetes.io/projected/5e0e0195-45b4-408f-a189-fd00ab3f0bb4-kube-api-access-bfmvs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:23.621112 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.621079 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" Sep 10 17:19:23.621462 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.621086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kp9cf" event={"ID":"5e0e0195-45b4-408f-a189-fd00ab3f0bb4","Type":"ContainerDied","Data":"0610ccf36b17bf90c2270b655e3f64cc747a651e0334f1a8f7d01733b9e16d71"} Sep 10 17:19:23.621462 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.621216 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0610ccf36b17bf90c2270b655e3f64cc747a651e0334f1a8f7d01733b9e16d71" Sep 10 17:19:23.775365 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.775340 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:23.804449 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.804425 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:23.807713 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.807688 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:23.927490 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927409 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w95ql\" (UniqueName: \"kubernetes.io/projected/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-kube-api-access-w95ql\") pod \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " Sep 10 17:19:23.927490 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927446 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-util\") pod \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " Sep 10 17:19:23.927490 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927469 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-bundle\") pod \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " Sep 10 17:19:23.927730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927536 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-bundle\") pod \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " Sep 10 17:19:23.927730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927563 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-util\") pod \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\" (UID: \"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4\") " Sep 10 17:19:23.927730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927589 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-bundle\") pod \"31ca1558-f25d-4f34-927d-0610e34b11d8\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " Sep 10 17:19:23.927730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927651 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flt9q\" (UniqueName: \"kubernetes.io/projected/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-kube-api-access-flt9q\") pod \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\" (UID: \"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50\") " Sep 10 17:19:23.927730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927679 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt7kk\" (UniqueName: \"kubernetes.io/projected/31ca1558-f25d-4f34-927d-0610e34b11d8-kube-api-access-bt7kk\") pod \"31ca1558-f25d-4f34-927d-0610e34b11d8\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " Sep 10 17:19:23.927730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.927704 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-util\") pod \"31ca1558-f25d-4f34-927d-0610e34b11d8\" (UID: \"31ca1558-f25d-4f34-927d-0610e34b11d8\") " Sep 10 17:19:23.928588 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.928163 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-bundle" (OuterVolumeSpecName: "bundle") pod "0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" (UID: "0d3f8071-4c37-4d8f-94f3-0a1b68d67c50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:23.928588 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.928369 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-bundle" (OuterVolumeSpecName: "bundle") pod "8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" (UID: "8f9b248e-c4c1-4f78-99e0-eda1cc28dde4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:23.928848 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.928685 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-bundle" (OuterVolumeSpecName: "bundle") pod "31ca1558-f25d-4f34-927d-0610e34b11d8" (UID: "31ca1558-f25d-4f34-927d-0610e34b11d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:23.930147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.930118 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-kube-api-access-w95ql" (OuterVolumeSpecName: "kube-api-access-w95ql") pod "8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" (UID: "8f9b248e-c4c1-4f78-99e0-eda1cc28dde4"). InnerVolumeSpecName "kube-api-access-w95ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:19:23.930357 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.930333 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ca1558-f25d-4f34-927d-0610e34b11d8-kube-api-access-bt7kk" (OuterVolumeSpecName: "kube-api-access-bt7kk") pod "31ca1558-f25d-4f34-927d-0610e34b11d8" (UID: "31ca1558-f25d-4f34-927d-0610e34b11d8"). InnerVolumeSpecName "kube-api-access-bt7kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:19:23.930427 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.930365 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-kube-api-access-flt9q" (OuterVolumeSpecName: "kube-api-access-flt9q") pod "0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" (UID: "0d3f8071-4c37-4d8f-94f3-0a1b68d67c50"). InnerVolumeSpecName "kube-api-access-flt9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:19:23.934445 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.934423 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-util" (OuterVolumeSpecName: "util") pod "31ca1558-f25d-4f34-927d-0610e34b11d8" (UID: "31ca1558-f25d-4f34-927d-0610e34b11d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:23.936049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.936005 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-util" (OuterVolumeSpecName: "util") pod "8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" (UID: "8f9b248e-c4c1-4f78-99e0-eda1cc28dde4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:23.936498 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:23.936471 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-util" (OuterVolumeSpecName: "util") pod "0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" (UID: "0d3f8071-4c37-4d8f-94f3-0a1b68d67c50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:24.028642 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028610 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flt9q\" (UniqueName: \"kubernetes.io/projected/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-kube-api-access-flt9q\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028642 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028638 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt7kk\" (UniqueName: \"kubernetes.io/projected/31ca1558-f25d-4f34-927d-0610e34b11d8-kube-api-access-bt7kk\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028652 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028667 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w95ql\" (UniqueName: \"kubernetes.io/projected/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-kube-api-access-w95ql\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028678 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028689 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d3f8071-4c37-4d8f-94f3-0a1b68d67c50-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028700 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028711 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9b248e-c4c1-4f78-99e0-eda1cc28dde4-util\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.028824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.028721 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca1558-f25d-4f34-927d-0610e34b11d8-bundle\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:24.628362 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.628331 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" Sep 10 17:19:24.628726 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.628338 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp6stk" event={"ID":"0d3f8071-4c37-4d8f-94f3-0a1b68d67c50","Type":"ContainerDied","Data":"30688dd83b53d86897d1a692b163ae44b318cccc8a9e71f0af8cf21c4217c3c1"} Sep 10 17:19:24.628726 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.628511 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30688dd83b53d86897d1a692b163ae44b318cccc8a9e71f0af8cf21c4217c3c1" Sep 10 17:19:24.630355 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.630328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" event={"ID":"31ca1558-f25d-4f34-927d-0610e34b11d8","Type":"ContainerDied","Data":"66e34b6baba8b7e426207ef186bf96dfb48aa2573d18c706c2c2e18bc138bc42"} Sep 10 17:19:24.630492 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.630363 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7nn8v" Sep 10 17:19:24.630492 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.630371 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e34b6baba8b7e426207ef186bf96dfb48aa2573d18c706c2c2e18bc138bc42" Sep 10 17:19:24.632780 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.632757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" event={"ID":"8f9b248e-c4c1-4f78-99e0-eda1cc28dde4","Type":"ContainerDied","Data":"542e761d61f188d61bd13118fe0eee4734f84a0e1b03be8f23ac83ef83975ad0"} Sep 10 17:19:24.632866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.632793 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="542e761d61f188d61bd13118fe0eee4734f84a0e1b03be8f23ac83ef83975ad0" Sep 10 17:19:24.632866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:24.632812 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vqp85" Sep 10 17:19:28.895414 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:28.895378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:28.998246 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:28.998207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:31.067700 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:31.067667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:31.455623 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:31.455531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:31.873939 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:31.873901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:34.131586 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:34.131550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:35.174555 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174520 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7"] Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174844 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerName="util" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174856 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerName="util" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174866 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerName="pull" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174871 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerName="pull" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174895 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerName="extract" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174902 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerName="extract" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174909 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerName="util" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174914 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerName="util" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174922 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerName="pull" Sep 10 17:19:35.174923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174927 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerName="pull" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174937 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerName="util" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174942 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerName="util" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174948 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174953 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174959 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerName="util" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174964 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerName="util" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174969 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerName="pull" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174974 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerName="pull" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174979 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174984 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174992 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerName="pull" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.174996 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerName="pull" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.175001 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.175005 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.175057 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e0e0195-45b4-408f-a189-fd00ab3f0bb4" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.175063 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ca1558-f25d-4f34-927d-0610e34b11d8" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.175069 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9b248e-c4c1-4f78-99e0-eda1cc28dde4" containerName="extract" Sep 10 17:19:35.175241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.175076 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3f8071-4c37-4d8f-94f3-0a1b68d67c50" containerName="extract" Sep 10 17:19:35.179130 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.179114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.182053 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.182030 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 17:19:35.182268 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.182253 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 17:19:35.183026 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.183012 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jfn2p\"" Sep 10 17:19:35.208453 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.208422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7"] Sep 10 17:19:35.225802 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.225766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66501440-f4ff-4298-a543-0dcba933e565-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.226000 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.225811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mb9d\" (UniqueName: \"kubernetes.io/projected/66501440-f4ff-4298-a543-0dcba933e565-kube-api-access-9mb9d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.241719 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.241686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:35.326850 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.326814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66501440-f4ff-4298-a543-0dcba933e565-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.327160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.326852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mb9d\" (UniqueName: \"kubernetes.io/projected/66501440-f4ff-4298-a543-0dcba933e565-kube-api-access-9mb9d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.327238 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.327181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66501440-f4ff-4298-a543-0dcba933e565-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.350192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.350160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mb9d\" (UniqueName: \"kubernetes.io/projected/66501440-f4ff-4298-a543-0dcba933e565-kube-api-access-9mb9d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.478099 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.478013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:35.489415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.489387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:35.756410 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.756329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:35.800532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:35.800503 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7"] Sep 10 17:19:35.802866 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:19:35.802838 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66501440_f4ff_4298_a543_0dcba933e565.slice/crio-f7ce4db6c38b1abac9e9fb22cc7032e9d1690f87b97ef3d314ebcb3081fe3da5 WatchSource:0}: Error finding container f7ce4db6c38b1abac9e9fb22cc7032e9d1690f87b97ef3d314ebcb3081fe3da5: Status 404 returned error can't find the container with id f7ce4db6c38b1abac9e9fb22cc7032e9d1690f87b97ef3d314ebcb3081fe3da5 Sep 10 17:19:36.698786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:36.698738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" event={"ID":"66501440-f4ff-4298-a543-0dcba933e565","Type":"ContainerStarted","Data":"f7ce4db6c38b1abac9e9fb22cc7032e9d1690f87b97ef3d314ebcb3081fe3da5"} Sep 10 17:19:37.351768 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:37.351728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:37.484281 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:37.484244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:38.079452 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:38.079396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:40.001023 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:40.000993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:40.136125 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:40.136088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:40.724135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:40.724094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" event={"ID":"66501440-f4ff-4298-a543-0dcba933e565","Type":"ContainerStarted","Data":"e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76"} Sep 10 17:19:40.724325 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:40.724208 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:40.747836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:40.747786 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" podStartSLOduration=1.770847485 podStartE2EDuration="5.747770048s" totalImagesPullingTime="3.976922563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:19:40.745429038 +0000 UTC m=+486.726509035" watchObservedRunningTime="2026-09-10 17:19:40.747770048 +0000 UTC m=+486.728850047" Sep 10 17:19:41.534520 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:41.534467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:41.660839 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:41.660802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:42.840994 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.840960 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf"] Sep 10 17:19:42.844469 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.844445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.848081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.848059 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 10 17:19:42.848200 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.848062 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 10 17:19:42.848200 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.848064 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-29zjg\"" Sep 10 17:19:42.853393 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.853367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:42.860353 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.860334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf"] Sep 10 17:19:42.884457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.884429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6e20548-80e2-43c7-8f64-66167f80e60f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.884610 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.884494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccnjc\" (UniqueName: \"kubernetes.io/projected/d6e20548-80e2-43c7-8f64-66167f80e60f-kube-api-access-ccnjc\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.884610 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.884585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6e20548-80e2-43c7-8f64-66167f80e60f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.985746 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.985713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6e20548-80e2-43c7-8f64-66167f80e60f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.985969 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.985765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6e20548-80e2-43c7-8f64-66167f80e60f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.985969 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.985803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccnjc\" (UniqueName: \"kubernetes.io/projected/d6e20548-80e2-43c7-8f64-66167f80e60f-kube-api-access-ccnjc\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.986468 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.986443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6e20548-80e2-43c7-8f64-66167f80e60f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.988189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.988156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6e20548-80e2-43c7-8f64-66167f80e60f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:42.989595 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:42.989567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:43.001628 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:43.001605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccnjc\" (UniqueName: \"kubernetes.io/projected/d6e20548-80e2-43c7-8f64-66167f80e60f-kube-api-access-ccnjc\") pod \"kuadrant-console-plugin-fbdd4f445-nlzbf\" (UID: \"d6e20548-80e2-43c7-8f64-66167f80e60f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:43.153943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:43.153821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" Sep 10 17:19:43.427465 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:43.427442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf"] Sep 10 17:19:43.429489 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:19:43.429461 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6e20548_80e2_43c7_8f64_66167f80e60f.slice/crio-db58d31e3af470cb95d4eda11722df67988de2dcfe4f41a38186ae93ed48b259 WatchSource:0}: Error finding container db58d31e3af470cb95d4eda11722df67988de2dcfe4f41a38186ae93ed48b259: Status 404 returned error can't find the container with id db58d31e3af470cb95d4eda11722df67988de2dcfe4f41a38186ae93ed48b259 Sep 10 17:19:43.586758 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:43.586719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:43.740800 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:43.740709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" event={"ID":"d6e20548-80e2-43c7-8f64-66167f80e60f","Type":"ContainerStarted","Data":"db58d31e3af470cb95d4eda11722df67988de2dcfe4f41a38186ae93ed48b259"} Sep 10 17:19:45.641842 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:45.641794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:45.716314 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:45.716268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:46.046907 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:46.046803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:46.148281 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:46.148213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:46.527834 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:46.527736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:46.729659 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:46.729625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:48.773251 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:48.773219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" event={"ID":"d6e20548-80e2-43c7-8f64-66167f80e60f","Type":"ContainerStarted","Data":"7fdd56be0dd7e7bdb9cfd56507800dd3c14d63e1d520a677d25161b9a18cecff"} Sep 10 17:19:51.732273 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:51.732243 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:51.773713 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:51.773664 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nlzbf" podStartSLOduration=4.993038552 podStartE2EDuration="9.77364673s" totalImagesPullingTime="4.780608178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:19:48.794801691 +0000 UTC m=+494.775881688" watchObservedRunningTime="2026-09-10 17:19:51.77364673 +0000 UTC m=+497.754726727" Sep 10 17:19:52.010938 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:52.010835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:52.122894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:52.122839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:54.142004 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.141966 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7"] Sep 10 17:19:54.142508 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.142178 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" podUID="66501440-f4ff-4298-a543-0dcba933e565" containerName="manager" containerID="cri-o://e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76" gracePeriod=2 Sep 10 17:19:54.158729 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.158483 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7"] Sep 10 17:19:54.160276 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:19:54.160242 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"66501440-f4ff-4298-a543-0dcba933e565\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\": pods \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:54.182191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.182157 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk"] Sep 10 17:19:54.182521 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.182508 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66501440-f4ff-4298-a543-0dcba933e565" containerName="manager" Sep 10 17:19:54.182521 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.182521 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="66501440-f4ff-4298-a543-0dcba933e565" containerName="manager" Sep 10 17:19:54.182614 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.182580 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="66501440-f4ff-4298-a543-0dcba933e565" containerName="manager" Sep 10 17:19:54.185841 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.185805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.189793 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:19:54.189749 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" is forbidden: User \"system:node:ip-10-0-137-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-217.ec2.internal' and this object" podUID="66501440-f4ff-4298-a543-0dcba933e565" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:54.205778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.205693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk"] Sep 10 17:19:54.296451 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.296410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnhh\" (UniqueName: \"kubernetes.io/projected/a3d1548f-e29a-427b-a025-e056863ee2ce-kube-api-access-vcnhh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h4ljk\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.296652 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.296469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3d1548f-e29a-427b-a025-e056863ee2ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h4ljk\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.389043 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.389015 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:54.392425 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:19:54.392365 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" is forbidden: User \"system:node:ip-10-0-137-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-217.ec2.internal' and this object" podUID="66501440-f4ff-4298-a543-0dcba933e565" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:54.397796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.397762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3d1548f-e29a-427b-a025-e056863ee2ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h4ljk\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.397963 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.397945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnhh\" (UniqueName: \"kubernetes.io/projected/a3d1548f-e29a-427b-a025-e056863ee2ce-kube-api-access-vcnhh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h4ljk\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.398192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.398173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3d1548f-e29a-427b-a025-e056863ee2ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h4ljk\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.412590 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.412557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnhh\" (UniqueName: \"kubernetes.io/projected/a3d1548f-e29a-427b-a025-e056863ee2ce-kube-api-access-vcnhh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-h4ljk\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.486819 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:19:54.486786 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ffgh7\" is forbidden: User \"system:node:ip-10-0-137-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-217.ec2.internal' and this object" podUID="66501440-f4ff-4298-a543-0dcba933e565" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:54.498793 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.498761 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66501440-f4ff-4298-a543-0dcba933e565-extensions-socket-volume\") pod \"66501440-f4ff-4298-a543-0dcba933e565\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " Sep 10 17:19:54.498998 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.498813 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mb9d\" (UniqueName: \"kubernetes.io/projected/66501440-f4ff-4298-a543-0dcba933e565-kube-api-access-9mb9d\") pod \"66501440-f4ff-4298-a543-0dcba933e565\" (UID: \"66501440-f4ff-4298-a543-0dcba933e565\") " Sep 10 17:19:54.499327 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.499302 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66501440-f4ff-4298-a543-0dcba933e565-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "66501440-f4ff-4298-a543-0dcba933e565" (UID: "66501440-f4ff-4298-a543-0dcba933e565"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:19:54.501048 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.501019 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66501440-f4ff-4298-a543-0dcba933e565-kube-api-access-9mb9d" (OuterVolumeSpecName: "kube-api-access-9mb9d") pod "66501440-f4ff-4298-a543-0dcba933e565" (UID: "66501440-f4ff-4298-a543-0dcba933e565"). InnerVolumeSpecName "kube-api-access-9mb9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:19:54.520215 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.520178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:54.600472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.600433 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66501440-f4ff-4298-a543-0dcba933e565-extensions-socket-volume\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:54.600472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.600474 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mb9d\" (UniqueName: \"kubernetes.io/projected/66501440-f4ff-4298-a543-0dcba933e565-kube-api-access-9mb9d\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:19:54.806182 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.806081 2576 generic.go:356] "Generic (PLEG): container finished" podID="66501440-f4ff-4298-a543-0dcba933e565" containerID="e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76" exitCode=0 Sep 10 17:19:54.806182 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.806145 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ffgh7" Sep 10 17:19:54.806401 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.806159 2576 scope.go:117] "RemoveContainer" containerID="e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76" Sep 10 17:19:54.808854 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.808828 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk"] Sep 10 17:19:54.810617 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:19:54.810595 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d1548f_e29a_427b_a025_e056863ee2ce.slice/crio-60057134cbb920bc6c525a3a7e6ec8a5186f334e0094d531555a1a2eaf5cfbfb WatchSource:0}: Error finding container 60057134cbb920bc6c525a3a7e6ec8a5186f334e0094d531555a1a2eaf5cfbfb: Status 404 returned error can't find the container with id 60057134cbb920bc6c525a3a7e6ec8a5186f334e0094d531555a1a2eaf5cfbfb Sep 10 17:19:54.815309 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.815208 2576 scope.go:117] "RemoveContainer" containerID="e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76" Sep 10 17:19:54.815643 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:19:54.815616 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76\": container with ID starting with e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76 not found: ID does not exist" containerID="e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76" Sep 10 17:19:54.815732 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:54.815656 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76"} err="failed to get container status \"e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76\": rpc error: code = NotFound desc = could not find container \"e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76\": container with ID starting with e3aad51efd718236c5e06c59aaae6bf7b69127aa289c3fb813a8083e7c24ec76 not found: ID does not exist" Sep 10 17:19:55.813941 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:55.813904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" event={"ID":"a3d1548f-e29a-427b-a025-e056863ee2ce","Type":"ContainerStarted","Data":"bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3"} Sep 10 17:19:55.814343 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:55.813958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" event={"ID":"a3d1548f-e29a-427b-a025-e056863ee2ce","Type":"ContainerStarted","Data":"60057134cbb920bc6c525a3a7e6ec8a5186f334e0094d531555a1a2eaf5cfbfb"} Sep 10 17:19:55.814343 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:55.814013 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:19:55.842181 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:55.842128 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" podStartSLOduration=1.842111872 podStartE2EDuration="1.842111872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:19:55.839425908 +0000 UTC m=+501.820505908" watchObservedRunningTime="2026-09-10 17:19:55.842111872 +0000 UTC m=+501.823191869" Sep 10 17:19:56.486320 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:56.486284 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66501440-f4ff-4298-a543-0dcba933e565" path="/var/lib/kubelet/pods/66501440-f4ff-4298-a543-0dcba933e565/volumes" Sep 10 17:19:58.481805 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:58.481763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:58.588109 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:58.588060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:19:58.690089 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:19:58.690039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:01.143803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:01.143764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:01.256808 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:01.256771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:04.972489 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:04.972452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:05.111623 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:05.111586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:05.214455 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:05.214418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:05.325602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:05.325574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:06.821476 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:06.821448 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:20:11.458807 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:11.458771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:11.628279 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:11.628244 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk"] Sep 10 17:20:11.628494 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:11.628470 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" podUID="a3d1548f-e29a-427b-a025-e056863ee2ce" containerName="manager" containerID="cri-o://bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3" gracePeriod=10 Sep 10 17:20:11.674770 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:11.674732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:12.567809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.567784 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:20:12.652917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.652829 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3d1548f-e29a-427b-a025-e056863ee2ce-extensions-socket-volume\") pod \"a3d1548f-e29a-427b-a025-e056863ee2ce\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " Sep 10 17:20:12.653063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.652962 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcnhh\" (UniqueName: \"kubernetes.io/projected/a3d1548f-e29a-427b-a025-e056863ee2ce-kube-api-access-vcnhh\") pod \"a3d1548f-e29a-427b-a025-e056863ee2ce\" (UID: \"a3d1548f-e29a-427b-a025-e056863ee2ce\") " Sep 10 17:20:12.653272 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.653246 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d1548f-e29a-427b-a025-e056863ee2ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a3d1548f-e29a-427b-a025-e056863ee2ce" (UID: "a3d1548f-e29a-427b-a025-e056863ee2ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:20:12.655135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.655113 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d1548f-e29a-427b-a025-e056863ee2ce-kube-api-access-vcnhh" (OuterVolumeSpecName: "kube-api-access-vcnhh") pod "a3d1548f-e29a-427b-a025-e056863ee2ce" (UID: "a3d1548f-e29a-427b-a025-e056863ee2ce"). InnerVolumeSpecName "kube-api-access-vcnhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:20:12.753735 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.753710 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcnhh\" (UniqueName: \"kubernetes.io/projected/a3d1548f-e29a-427b-a025-e056863ee2ce-kube-api-access-vcnhh\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:20:12.753735 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.753734 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3d1548f-e29a-427b-a025-e056863ee2ce-extensions-socket-volume\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:20:12.904548 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.904465 2576 generic.go:356] "Generic (PLEG): container finished" podID="a3d1548f-e29a-427b-a025-e056863ee2ce" containerID="bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3" exitCode=0 Sep 10 17:20:12.904548 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.904511 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" Sep 10 17:20:12.904548 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.904517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" event={"ID":"a3d1548f-e29a-427b-a025-e056863ee2ce","Type":"ContainerDied","Data":"bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3"} Sep 10 17:20:12.904743 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.904562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk" event={"ID":"a3d1548f-e29a-427b-a025-e056863ee2ce","Type":"ContainerDied","Data":"60057134cbb920bc6c525a3a7e6ec8a5186f334e0094d531555a1a2eaf5cfbfb"} Sep 10 17:20:12.904743 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.904585 2576 scope.go:117] "RemoveContainer" containerID="bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3" Sep 10 17:20:12.914584 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.914569 2576 scope.go:117] "RemoveContainer" containerID="bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3" Sep 10 17:20:12.914854 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:20:12.914835 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3\": container with ID starting with bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3 not found: ID does not exist" containerID="bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3" Sep 10 17:20:12.914918 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.914863 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3"} err="failed to get container status \"bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3\": rpc error: code = NotFound desc = could not find container \"bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3\": container with ID starting with bf959c4da315c22550236a43c57d4f5c31215d624b52c0d2d43a386dd37634c3 not found: ID does not exist" Sep 10 17:20:12.932695 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.932673 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk"] Sep 10 17:20:12.946736 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:12.946716 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-h4ljk"] Sep 10 17:20:14.487329 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:14.487290 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d1548f-e29a-427b-a025-e056863ee2ce" path="/var/lib/kubelet/pods/a3d1548f-e29a-427b-a025-e056863ee2ce/volumes" Sep 10 17:20:17.934205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:17.934163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:18.048832 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:18.048798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:21.389262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:21.389216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:21.506670 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:21.506631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:28.233937 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.233843 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd"] Sep 10 17:20:28.234274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.234193 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d1548f-e29a-427b-a025-e056863ee2ce" containerName="manager" Sep 10 17:20:28.234274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.234204 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d1548f-e29a-427b-a025-e056863ee2ce" containerName="manager" Sep 10 17:20:28.234274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.234263 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d1548f-e29a-427b-a025-e056863ee2ce" containerName="manager" Sep 10 17:20:28.253667 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.253644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.259615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.259586 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sgwlf\"" Sep 10 17:20:28.263356 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.263332 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd"] Sep 10 17:20:28.296143 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.296113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:28.384341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.384609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.384592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxmx\" (UniqueName: \"kubernetes.io/projected/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-kube-api-access-rgxmx\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.485908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.485780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxmx\" (UniqueName: \"kubernetes.io/projected/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-kube-api-access-rgxmx\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.486198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.486012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.486198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.486078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.486198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.486092 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.486306 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.486275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.486444 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.486416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.488141 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.488119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.488369 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.488350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.500242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.500218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxmx\" (UniqueName: \"kubernetes.io/projected/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-kube-api-access-rgxmx\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.501072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.501052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a35ba9e-05d0-4a6a-90dc-03e621ea9e04-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gmhd\" (UID: \"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.565313 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.565280 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:28.628243 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.628210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:28.879169 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:20:28.879142 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a35ba9e_05d0_4a6a_90dc_03e621ea9e04.slice/crio-1d94ee96721d5b4a9c6489977e26aba2bcb5eb0e276f1f334b25461c7f36a6da WatchSource:0}: Error finding container 1d94ee96721d5b4a9c6489977e26aba2bcb5eb0e276f1f334b25461c7f36a6da: Status 404 returned error can't find the container with id 1d94ee96721d5b4a9c6489977e26aba2bcb5eb0e276f1f334b25461c7f36a6da Sep 10 17:20:28.879277 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.879255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd"] Sep 10 17:20:28.881314 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.881280 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:20:28.881406 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.881346 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:20:28.881406 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.881375 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:20:28.991208 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.991174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" event={"ID":"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04","Type":"ContainerStarted","Data":"448c4339ae9734d418b1f50f04cbef4251a539f052729564d5bf0161eae93b33"} Sep 10 17:20:28.991365 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:28.991233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" event={"ID":"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04","Type":"ContainerStarted","Data":"1d94ee96721d5b4a9c6489977e26aba2bcb5eb0e276f1f334b25461c7f36a6da"} Sep 10 17:20:29.037134 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:29.037077 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" podStartSLOduration=1.037061932 podStartE2EDuration="1.037061932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:29.036537118 +0000 UTC m=+535.017617115" watchObservedRunningTime="2026-09-10 17:20:29.037061932 +0000 UTC m=+535.018141931" Sep 10 17:20:29.566273 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:29.566241 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:29.571210 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:29.571183 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:29.996900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:29.996794 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:29.997675 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:29.997656 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:20:30.216080 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.216047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:30.230486 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.230457 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2m2bh"] Sep 10 17:20:30.233790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.233776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.236567 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.236550 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 17:20:30.249361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.249302 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2m2bh"] Sep 10 17:20:30.334209 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.334179 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2m2bh"] Sep 10 17:20:30.385388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.385351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:30.401564 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.401529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2nb\" (UniqueName: \"kubernetes.io/projected/9856930f-e257-4bab-96ca-4eec3009f0b2-kube-api-access-hp2nb\") pod \"limitador-limitador-8588749977-2m2bh\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.401698 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.401573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9856930f-e257-4bab-96ca-4eec3009f0b2-config-file\") pod \"limitador-limitador-8588749977-2m2bh\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.502136 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.502058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2nb\" (UniqueName: \"kubernetes.io/projected/9856930f-e257-4bab-96ca-4eec3009f0b2-kube-api-access-hp2nb\") pod \"limitador-limitador-8588749977-2m2bh\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.502136 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.502094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9856930f-e257-4bab-96ca-4eec3009f0b2-config-file\") pod \"limitador-limitador-8588749977-2m2bh\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.502718 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.502698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9856930f-e257-4bab-96ca-4eec3009f0b2-config-file\") pod \"limitador-limitador-8588749977-2m2bh\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.508645 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.508615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:30.518465 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.518444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2nb\" (UniqueName: \"kubernetes.io/projected/9856930f-e257-4bab-96ca-4eec3009f0b2-kube-api-access-hp2nb\") pod \"limitador-limitador-8588749977-2m2bh\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.544848 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.544826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:30.848331 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:30.848305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2m2bh"] Sep 10 17:20:30.850051 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:20:30.850025 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9856930f_e257_4bab_96ca_4eec3009f0b2.slice/crio-e889b2d93defd77911cb1fd5e020d3cfa1b109ff6356fd04e206ae9b1d04fcf5 WatchSource:0}: Error finding container e889b2d93defd77911cb1fd5e020d3cfa1b109ff6356fd04e206ae9b1d04fcf5: Status 404 returned error can't find the container with id e889b2d93defd77911cb1fd5e020d3cfa1b109ff6356fd04e206ae9b1d04fcf5 Sep 10 17:20:31.002769 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.002733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" event={"ID":"9856930f-e257-4bab-96ca-4eec3009f0b2","Type":"ContainerStarted","Data":"e889b2d93defd77911cb1fd5e020d3cfa1b109ff6356fd04e206ae9b1d04fcf5"} Sep 10 17:20:31.418824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.418793 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q8xwx"] Sep 10 17:20:31.423426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.423403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" Sep 10 17:20:31.430348 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.430328 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vbl9d\"" Sep 10 17:20:31.442371 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.442350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q8xwx"] Sep 10 17:20:31.548237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.548204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:31.612116 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.612092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfd4\" (UniqueName: \"kubernetes.io/projected/af727c04-2510-4dcd-a0f2-b5eb60bd586d-kube-api-access-2rfd4\") pod \"authorino-7bcfd8f9cd-q8xwx\" (UID: \"af727c04-2510-4dcd-a0f2-b5eb60bd586d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" Sep 10 17:20:31.703262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.703183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:31.713234 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.713205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfd4\" (UniqueName: \"kubernetes.io/projected/af727c04-2510-4dcd-a0f2-b5eb60bd586d-kube-api-access-2rfd4\") pod \"authorino-7bcfd8f9cd-q8xwx\" (UID: \"af727c04-2510-4dcd-a0f2-b5eb60bd586d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" Sep 10 17:20:31.730624 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.730595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfd4\" (UniqueName: \"kubernetes.io/projected/af727c04-2510-4dcd-a0f2-b5eb60bd586d-kube-api-access-2rfd4\") pod \"authorino-7bcfd8f9cd-q8xwx\" (UID: \"af727c04-2510-4dcd-a0f2-b5eb60bd586d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" Sep 10 17:20:31.732253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:31.732228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" Sep 10 17:20:32.075005 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:20:32.074979 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf727c04_2510_4dcd_a0f2_b5eb60bd586d.slice/crio-ca20d3229ed20a97145250c5e0761e72b3ca1aafb27604d51d8aef812e30a1b6 WatchSource:0}: Error finding container ca20d3229ed20a97145250c5e0761e72b3ca1aafb27604d51d8aef812e30a1b6: Status 404 returned error can't find the container with id ca20d3229ed20a97145250c5e0761e72b3ca1aafb27604d51d8aef812e30a1b6 Sep 10 17:20:32.075590 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:32.075564 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q8xwx"] Sep 10 17:20:33.014213 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:33.014180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" event={"ID":"af727c04-2510-4dcd-a0f2-b5eb60bd586d","Type":"ContainerStarted","Data":"ca20d3229ed20a97145250c5e0761e72b3ca1aafb27604d51d8aef812e30a1b6"} Sep 10 17:20:33.015792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:33.015760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" event={"ID":"9856930f-e257-4bab-96ca-4eec3009f0b2","Type":"ContainerStarted","Data":"c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82"} Sep 10 17:20:33.015978 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:33.015863 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:33.042205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:33.042158 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" podStartSLOduration=1.891505859 podStartE2EDuration="3.042142317s" totalImagesPullingTime="1.150636458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:33.041731627 +0000 UTC m=+539.022811624" watchObservedRunningTime="2026-09-10 17:20:33.042142317 +0000 UTC m=+539.023222314" Sep 10 17:20:34.714610 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:34.714579 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q8xwx"] Sep 10 17:20:35.029249 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:35.029212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" event={"ID":"af727c04-2510-4dcd-a0f2-b5eb60bd586d","Type":"ContainerStarted","Data":"82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435"} Sep 10 17:20:35.056439 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:35.056390 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" podStartSLOduration=2.022985084 podStartE2EDuration="4.056373233s" totalImagesPullingTime="2.033388149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:35.054381063 +0000 UTC m=+541.035461062" watchObservedRunningTime="2026-09-10 17:20:35.056373233 +0000 UTC m=+541.037453230" Sep 10 17:20:36.034599 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:36.034561 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" podUID="af727c04-2510-4dcd-a0f2-b5eb60bd586d" containerName="authorino" containerID="cri-o://82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435" gracePeriod=30 Sep 10 17:20:36.278250 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:36.278227 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" Sep 10 17:20:36.355981 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:36.355953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rfd4\" (UniqueName: \"kubernetes.io/projected/af727c04-2510-4dcd-a0f2-b5eb60bd586d-kube-api-access-2rfd4\") pod \"af727c04-2510-4dcd-a0f2-b5eb60bd586d\" (UID: \"af727c04-2510-4dcd-a0f2-b5eb60bd586d\") " Sep 10 17:20:36.357952 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:36.357926 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af727c04-2510-4dcd-a0f2-b5eb60bd586d-kube-api-access-2rfd4" (OuterVolumeSpecName: "kube-api-access-2rfd4") pod "af727c04-2510-4dcd-a0f2-b5eb60bd586d" (UID: "af727c04-2510-4dcd-a0f2-b5eb60bd586d"). InnerVolumeSpecName "kube-api-access-2rfd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:20:36.457648 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:36.457616 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfd4\" (UniqueName: \"kubernetes.io/projected/af727c04-2510-4dcd-a0f2-b5eb60bd586d-kube-api-access-2rfd4\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:20:37.040619 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.040588 2576 generic.go:356] "Generic (PLEG): container finished" podID="af727c04-2510-4dcd-a0f2-b5eb60bd586d" containerID="82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435" exitCode=0 Sep 10 17:20:37.041051 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.040628 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" Sep 10 17:20:37.041051 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.040655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" event={"ID":"af727c04-2510-4dcd-a0f2-b5eb60bd586d","Type":"ContainerDied","Data":"82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435"} Sep 10 17:20:37.041051 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.040685 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-q8xwx" event={"ID":"af727c04-2510-4dcd-a0f2-b5eb60bd586d","Type":"ContainerDied","Data":"ca20d3229ed20a97145250c5e0761e72b3ca1aafb27604d51d8aef812e30a1b6"} Sep 10 17:20:37.041051 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.040701 2576 scope.go:117] "RemoveContainer" containerID="82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435" Sep 10 17:20:37.049258 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.049239 2576 scope.go:117] "RemoveContainer" containerID="82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435" Sep 10 17:20:37.049541 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:20:37.049520 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435\": container with ID starting with 82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435 not found: ID does not exist" containerID="82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435" Sep 10 17:20:37.049594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.049549 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435"} err="failed to get container status \"82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435\": rpc error: code = NotFound desc = could not find container \"82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435\": container with ID starting with 82ef3b4a9881307a5ed5d793b86bbe0d4b1d9c916d8ff93258bfca52d0c03435 not found: ID does not exist" Sep 10 17:20:37.075602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.075576 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q8xwx"] Sep 10 17:20:37.079951 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:37.079928 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-q8xwx"] Sep 10 17:20:38.486491 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:38.486459 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af727c04-2510-4dcd-a0f2-b5eb60bd586d" path="/var/lib/kubelet/pods/af727c04-2510-4dcd-a0f2-b5eb60bd586d/volumes" Sep 10 17:20:43.364661 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:43.364628 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2m2bh"] Sep 10 17:20:43.365081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:43.364821 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" podUID="9856930f-e257-4bab-96ca-4eec3009f0b2" containerName="limitador" containerID="cri-o://c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82" gracePeriod=30 Sep 10 17:20:43.365576 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:43.365491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:43.913189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:43.913164 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:44.023461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.023386 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp2nb\" (UniqueName: \"kubernetes.io/projected/9856930f-e257-4bab-96ca-4eec3009f0b2-kube-api-access-hp2nb\") pod \"9856930f-e257-4bab-96ca-4eec3009f0b2\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " Sep 10 17:20:44.023461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.023437 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9856930f-e257-4bab-96ca-4eec3009f0b2-config-file\") pod \"9856930f-e257-4bab-96ca-4eec3009f0b2\" (UID: \"9856930f-e257-4bab-96ca-4eec3009f0b2\") " Sep 10 17:20:44.023798 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.023766 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9856930f-e257-4bab-96ca-4eec3009f0b2-config-file" (OuterVolumeSpecName: "config-file") pod "9856930f-e257-4bab-96ca-4eec3009f0b2" (UID: "9856930f-e257-4bab-96ca-4eec3009f0b2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:20:44.025380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.025359 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9856930f-e257-4bab-96ca-4eec3009f0b2-kube-api-access-hp2nb" (OuterVolumeSpecName: "kube-api-access-hp2nb") pod "9856930f-e257-4bab-96ca-4eec3009f0b2" (UID: "9856930f-e257-4bab-96ca-4eec3009f0b2"). InnerVolumeSpecName "kube-api-access-hp2nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:20:44.077398 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.077363 2576 generic.go:356] "Generic (PLEG): container finished" podID="9856930f-e257-4bab-96ca-4eec3009f0b2" containerID="c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82" exitCode=0 Sep 10 17:20:44.077535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.077408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" event={"ID":"9856930f-e257-4bab-96ca-4eec3009f0b2","Type":"ContainerDied","Data":"c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82"} Sep 10 17:20:44.077535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.077449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" event={"ID":"9856930f-e257-4bab-96ca-4eec3009f0b2","Type":"ContainerDied","Data":"e889b2d93defd77911cb1fd5e020d3cfa1b109ff6356fd04e206ae9b1d04fcf5"} Sep 10 17:20:44.077535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.077470 2576 scope.go:117] "RemoveContainer" containerID="c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82" Sep 10 17:20:44.077535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.077412 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2m2bh" Sep 10 17:20:44.086645 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.086605 2576 scope.go:117] "RemoveContainer" containerID="c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82" Sep 10 17:20:44.086872 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:20:44.086853 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82\": container with ID starting with c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82 not found: ID does not exist" containerID="c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82" Sep 10 17:20:44.086990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.086904 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82"} err="failed to get container status \"c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82\": rpc error: code = NotFound desc = could not find container \"c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82\": container with ID starting with c548e3c3be7811f7bcc139c3183640b0f61f2a234e3069cf5e0e7be207ab8d82 not found: ID does not exist" Sep 10 17:20:44.109786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.109765 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2m2bh"] Sep 10 17:20:44.116245 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.116220 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2m2bh"] Sep 10 17:20:44.124493 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.124473 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp2nb\" (UniqueName: \"kubernetes.io/projected/9856930f-e257-4bab-96ca-4eec3009f0b2-kube-api-access-hp2nb\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:20:44.124493 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.124491 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9856930f-e257-4bab-96ca-4eec3009f0b2-config-file\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:20:44.487356 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:44.487325 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9856930f-e257-4bab-96ca-4eec3009f0b2" path="/var/lib/kubelet/pods/9856930f-e257-4bab-96ca-4eec3009f0b2/volumes" Sep 10 17:20:46.387376 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.387344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:46.606462 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.606427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:46.876716 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.876679 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn"] Sep 10 17:20:46.877108 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.877091 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9856930f-e257-4bab-96ca-4eec3009f0b2" containerName="limitador" Sep 10 17:20:46.877207 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.877110 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9856930f-e257-4bab-96ca-4eec3009f0b2" containerName="limitador" Sep 10 17:20:46.877207 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.877121 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af727c04-2510-4dcd-a0f2-b5eb60bd586d" containerName="authorino" Sep 10 17:20:46.877207 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.877128 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af727c04-2510-4dcd-a0f2-b5eb60bd586d" containerName="authorino" Sep 10 17:20:46.877359 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.877222 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9856930f-e257-4bab-96ca-4eec3009f0b2" containerName="limitador" Sep 10 17:20:46.877359 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.877238 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="af727c04-2510-4dcd-a0f2-b5eb60bd586d" containerName="authorino" Sep 10 17:20:46.881529 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.881509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:46.885671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.885652 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 10 17:20:46.886055 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.886038 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 17:20:46.886155 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.886116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8mrm4\"" Sep 10 17:20:46.886155 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.886126 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 17:20:46.910512 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.910480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn"] Sep 10 17:20:46.946234 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.946208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rq9f\" (UniqueName: \"kubernetes.io/projected/31896f7c-9e3e-46fc-973f-f5025b3afe88-kube-api-access-2rq9f\") pod \"postgres-69fb4594cf-vzqqn\" (UID: \"31896f7c-9e3e-46fc-973f-f5025b3afe88\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:46.946350 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.946267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31896f7c-9e3e-46fc-973f-f5025b3afe88-data\") pod \"postgres-69fb4594cf-vzqqn\" (UID: \"31896f7c-9e3e-46fc-973f-f5025b3afe88\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:46.949981 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:46.949956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:47.047064 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:47.047034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rq9f\" (UniqueName: \"kubernetes.io/projected/31896f7c-9e3e-46fc-973f-f5025b3afe88-kube-api-access-2rq9f\") pod \"postgres-69fb4594cf-vzqqn\" (UID: \"31896f7c-9e3e-46fc-973f-f5025b3afe88\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:47.047227 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:47.047079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31896f7c-9e3e-46fc-973f-f5025b3afe88-data\") pod \"postgres-69fb4594cf-vzqqn\" (UID: \"31896f7c-9e3e-46fc-973f-f5025b3afe88\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:47.047408 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:47.047394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31896f7c-9e3e-46fc-973f-f5025b3afe88-data\") pod \"postgres-69fb4594cf-vzqqn\" (UID: \"31896f7c-9e3e-46fc-973f-f5025b3afe88\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:47.060422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:47.060398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rq9f\" (UniqueName: \"kubernetes.io/projected/31896f7c-9e3e-46fc-973f-f5025b3afe88-kube-api-access-2rq9f\") pod \"postgres-69fb4594cf-vzqqn\" (UID: \"31896f7c-9e3e-46fc-973f-f5025b3afe88\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:47.191150 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:47.191053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:47.477310 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:47.477287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn"] Sep 10 17:20:47.479075 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:20:47.479045 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31896f7c_9e3e_46fc_973f_f5025b3afe88.slice/crio-9d4fbad2d6f6e33c1bc9f87a02575bbe70e3b65beaf367351c1886fc58f43d51 WatchSource:0}: Error finding container 9d4fbad2d6f6e33c1bc9f87a02575bbe70e3b65beaf367351c1886fc58f43d51: Status 404 returned error can't find the container with id 9d4fbad2d6f6e33c1bc9f87a02575bbe70e3b65beaf367351c1886fc58f43d51 Sep 10 17:20:48.099580 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:48.099541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" event={"ID":"31896f7c-9e3e-46fc-973f-f5025b3afe88","Type":"ContainerStarted","Data":"9d4fbad2d6f6e33c1bc9f87a02575bbe70e3b65beaf367351c1886fc58f43d51"} Sep 10 17:20:50.112057 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:50.112015 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" event={"ID":"31896f7c-9e3e-46fc-973f-f5025b3afe88","Type":"ContainerStarted","Data":"9451f9054c1a1f4ca12abd7fba813e176b959c210b6eeea47f2d0cd6f3df35e0"} Sep 10 17:20:50.112423 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:50.112085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:56.146551 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.146522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" Sep 10 17:20:56.188620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.188570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vzqqn" podStartSLOduration=7.976292348 podStartE2EDuration="10.188555152s" totalImagesPullingTime="2.212262804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:50.167079193 +0000 UTC m=+556.148159194" watchObservedRunningTime="2026-09-10 17:20:56.188555152 +0000 UTC m=+562.169635150" Sep 10 17:20:56.600733 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.600703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:56.849584 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.849552 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-cl976"] Sep 10 17:20:56.855491 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.855445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cl976" Sep 10 17:20:56.861851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.861701 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vbl9d\"" Sep 10 17:20:56.871061 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.871039 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-cl976"] Sep 10 17:20:56.930791 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:56.930762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q82mx\" (UniqueName: \"kubernetes.io/projected/ca25a0d9-371a-4f44-8d80-0d618eac7f01-kube-api-access-q82mx\") pod \"authorino-557f876f69-cl976\" (UID: \"ca25a0d9-371a-4f44-8d80-0d618eac7f01\") " pod="kuadrant-system/authorino-557f876f69-cl976" Sep 10 17:20:57.031804 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.031763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q82mx\" (UniqueName: \"kubernetes.io/projected/ca25a0d9-371a-4f44-8d80-0d618eac7f01-kube-api-access-q82mx\") pod \"authorino-557f876f69-cl976\" (UID: \"ca25a0d9-371a-4f44-8d80-0d618eac7f01\") " pod="kuadrant-system/authorino-557f876f69-cl976" Sep 10 17:20:57.046408 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.046382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q82mx\" (UniqueName: \"kubernetes.io/projected/ca25a0d9-371a-4f44-8d80-0d618eac7f01-kube-api-access-q82mx\") pod \"authorino-557f876f69-cl976\" (UID: \"ca25a0d9-371a-4f44-8d80-0d618eac7f01\") " pod="kuadrant-system/authorino-557f876f69-cl976" Sep 10 17:20:57.099338 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.099305 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-cl976"] Sep 10 17:20:57.099595 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.099582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cl976" Sep 10 17:20:57.129573 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.129540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:20:57.196498 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.196464 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7644879c59-79whx"] Sep 10 17:20:57.202594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.202571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7644879c59-79whx" Sep 10 17:20:57.218516 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.218492 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7644879c59-79whx"] Sep 10 17:20:57.233528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.233499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6c7k\" (UniqueName: \"kubernetes.io/projected/b3f4b084-19bc-46d3-ba8e-6c4661c49c01-kube-api-access-z6c7k\") pod \"authorino-7644879c59-79whx\" (UID: \"b3f4b084-19bc-46d3-ba8e-6c4661c49c01\") " pod="kuadrant-system/authorino-7644879c59-79whx" Sep 10 17:20:57.294074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.294049 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7644879c59-79whx"] Sep 10 17:20:57.294322 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:20:57.294304 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z6c7k], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7644879c59-79whx" podUID="b3f4b084-19bc-46d3-ba8e-6c4661c49c01" Sep 10 17:20:57.334945 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.334917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6c7k\" (UniqueName: \"kubernetes.io/projected/b3f4b084-19bc-46d3-ba8e-6c4661c49c01-kube-api-access-z6c7k\") pod \"authorino-7644879c59-79whx\" (UID: \"b3f4b084-19bc-46d3-ba8e-6c4661c49c01\") " pod="kuadrant-system/authorino-7644879c59-79whx" Sep 10 17:20:57.355235 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.355209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6c7k\" (UniqueName: \"kubernetes.io/projected/b3f4b084-19bc-46d3-ba8e-6c4661c49c01-kube-api-access-z6c7k\") pod \"authorino-7644879c59-79whx\" (UID: \"b3f4b084-19bc-46d3-ba8e-6c4661c49c01\") " pod="kuadrant-system/authorino-7644879c59-79whx" Sep 10 17:20:57.396683 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.396620 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f4c699d65-4f4kx"] Sep 10 17:20:57.401039 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.401022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:57.405694 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.405669 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 17:20:57.409827 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.409804 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-cl976"] Sep 10 17:20:57.411388 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:20:57.411368 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca25a0d9_371a_4f44_8d80_0d618eac7f01.slice/crio-bde99c174c1a6ea3a0226b0d9e24fba87c5b446fe74e7ec333d05d73f7060621 WatchSource:0}: Error finding container bde99c174c1a6ea3a0226b0d9e24fba87c5b446fe74e7ec333d05d73f7060621: Status 404 returned error can't find the container with id bde99c174c1a6ea3a0226b0d9e24fba87c5b446fe74e7ec333d05d73f7060621 Sep 10 17:20:57.435363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.435339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-tls-cert\") pod \"authorino-5f4c699d65-4f4kx\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:57.435475 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.435374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpq5\" (UniqueName: \"kubernetes.io/projected/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-kube-api-access-rgpq5\") pod \"authorino-5f4c699d65-4f4kx\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:57.439201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.439176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4c699d65-4f4kx"] Sep 10 17:20:57.536491 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.536459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-tls-cert\") pod \"authorino-5f4c699d65-4f4kx\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:57.536491 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.536496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpq5\" (UniqueName: \"kubernetes.io/projected/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-kube-api-access-rgpq5\") pod \"authorino-5f4c699d65-4f4kx\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:57.538900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.538863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-tls-cert\") pod \"authorino-5f4c699d65-4f4kx\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:57.550059 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.550032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpq5\" (UniqueName: \"kubernetes.io/projected/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-kube-api-access-rgpq5\") pod \"authorino-5f4c699d65-4f4kx\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:57.711170 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:57.711082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:20:58.002665 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.002587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4c699d65-4f4kx"] Sep 10 17:20:58.007760 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:20:58.007734 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce9c3f12_15a7_4362_adb2_b8eb423d6d33.slice/crio-dd59d6fb4eaca632cabaf189ee1ecbd1ec4740c296874933a184431a49327c65 WatchSource:0}: Error finding container dd59d6fb4eaca632cabaf189ee1ecbd1ec4740c296874933a184431a49327c65: Status 404 returned error can't find the container with id dd59d6fb4eaca632cabaf189ee1ecbd1ec4740c296874933a184431a49327c65 Sep 10 17:20:58.176376 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.176340 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" event={"ID":"ce9c3f12-15a7-4362-adb2-b8eb423d6d33","Type":"ContainerStarted","Data":"dd59d6fb4eaca632cabaf189ee1ecbd1ec4740c296874933a184431a49327c65"} Sep 10 17:20:58.177552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.177526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cl976" event={"ID":"ca25a0d9-371a-4f44-8d80-0d618eac7f01","Type":"ContainerStarted","Data":"bde99c174c1a6ea3a0226b0d9e24fba87c5b446fe74e7ec333d05d73f7060621"} Sep 10 17:20:58.177662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.177588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7644879c59-79whx" Sep 10 17:20:58.186908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.186873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7644879c59-79whx" Sep 10 17:20:58.241807 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.241782 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6c7k\" (UniqueName: \"kubernetes.io/projected/b3f4b084-19bc-46d3-ba8e-6c4661c49c01-kube-api-access-z6c7k\") pod \"b3f4b084-19bc-46d3-ba8e-6c4661c49c01\" (UID: \"b3f4b084-19bc-46d3-ba8e-6c4661c49c01\") " Sep 10 17:20:58.243825 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.243797 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f4b084-19bc-46d3-ba8e-6c4661c49c01-kube-api-access-z6c7k" (OuterVolumeSpecName: "kube-api-access-z6c7k") pod "b3f4b084-19bc-46d3-ba8e-6c4661c49c01" (UID: "b3f4b084-19bc-46d3-ba8e-6c4661c49c01"). InnerVolumeSpecName "kube-api-access-z6c7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:20:58.343237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:58.343200 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6c7k\" (UniqueName: \"kubernetes.io/projected/b3f4b084-19bc-46d3-ba8e-6c4661c49c01-kube-api-access-z6c7k\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:20:59.184856 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.184824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" event={"ID":"ce9c3f12-15a7-4362-adb2-b8eb423d6d33","Type":"ContainerStarted","Data":"af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84"} Sep 10 17:20:59.186367 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.186331 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-cl976" podUID="ca25a0d9-371a-4f44-8d80-0d618eac7f01" containerName="authorino" containerID="cri-o://8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0" gracePeriod=30 Sep 10 17:20:59.186367 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.186359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cl976" event={"ID":"ca25a0d9-371a-4f44-8d80-0d618eac7f01","Type":"ContainerStarted","Data":"8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0"} Sep 10 17:20:59.186556 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.186348 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7644879c59-79whx" Sep 10 17:20:59.217442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.217397 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" podStartSLOduration=1.848333811 podStartE2EDuration="2.217379348s" totalImagesPullingTime="369.045537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:59.217132859 +0000 UTC m=+565.198212857" watchObservedRunningTime="2026-09-10 17:20:59.217379348 +0000 UTC m=+565.198459385" Sep 10 17:20:59.282591 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.282559 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7644879c59-79whx"] Sep 10 17:20:59.288814 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.288789 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7644879c59-79whx"] Sep 10 17:20:59.343086 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.343036 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-cl976" podStartSLOduration=2.498438642 podStartE2EDuration="3.343018821s" totalImagesPullingTime="844.580179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:59.341904971 +0000 UTC m=+565.322984968" watchObservedRunningTime="2026-09-10 17:20:59.343018821 +0000 UTC m=+565.324098857" Sep 10 17:20:59.431978 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.431955 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cl976" Sep 10 17:20:59.555438 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.555354 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q82mx\" (UniqueName: \"kubernetes.io/projected/ca25a0d9-371a-4f44-8d80-0d618eac7f01-kube-api-access-q82mx\") pod \"ca25a0d9-371a-4f44-8d80-0d618eac7f01\" (UID: \"ca25a0d9-371a-4f44-8d80-0d618eac7f01\") " Sep 10 17:20:59.557415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.557390 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca25a0d9-371a-4f44-8d80-0d618eac7f01-kube-api-access-q82mx" (OuterVolumeSpecName: "kube-api-access-q82mx") pod "ca25a0d9-371a-4f44-8d80-0d618eac7f01" (UID: "ca25a0d9-371a-4f44-8d80-0d618eac7f01"). InnerVolumeSpecName "kube-api-access-q82mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:20:59.656025 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:20:59.655984 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q82mx\" (UniqueName: \"kubernetes.io/projected/ca25a0d9-371a-4f44-8d80-0d618eac7f01-kube-api-access-q82mx\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:21:00.192965 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.192925 2576 generic.go:356] "Generic (PLEG): container finished" podID="ca25a0d9-371a-4f44-8d80-0d618eac7f01" containerID="8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0" exitCode=0 Sep 10 17:21:00.193135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.192971 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cl976" Sep 10 17:21:00.193135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.193012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cl976" event={"ID":"ca25a0d9-371a-4f44-8d80-0d618eac7f01","Type":"ContainerDied","Data":"8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0"} Sep 10 17:21:00.193135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.193055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cl976" event={"ID":"ca25a0d9-371a-4f44-8d80-0d618eac7f01","Type":"ContainerDied","Data":"bde99c174c1a6ea3a0226b0d9e24fba87c5b446fe74e7ec333d05d73f7060621"} Sep 10 17:21:00.193135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.193071 2576 scope.go:117] "RemoveContainer" containerID="8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0" Sep 10 17:21:00.202301 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.202286 2576 scope.go:117] "RemoveContainer" containerID="8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0" Sep 10 17:21:00.202545 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:21:00.202528 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0\": container with ID starting with 8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0 not found: ID does not exist" containerID="8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0" Sep 10 17:21:00.202623 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.202553 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0"} err="failed to get container status \"8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0\": rpc error: code = NotFound desc = could not find container \"8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0\": container with ID starting with 8ed12f76097762bf112078c1fdb6699ae3b9a4a28aeff10a8f0555ade57c9db0 not found: ID does not exist" Sep 10 17:21:00.256041 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.256005 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-cl976"] Sep 10 17:21:00.259892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.259858 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-cl976"] Sep 10 17:21:00.486940 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.486860 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f4b084-19bc-46d3-ba8e-6c4661c49c01" path="/var/lib/kubelet/pods/b3f4b084-19bc-46d3-ba8e-6c4661c49c01/volumes" Sep 10 17:21:00.487273 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:00.487100 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca25a0d9-371a-4f44-8d80-0d618eac7f01" path="/var/lib/kubelet/pods/ca25a0d9-371a-4f44-8d80-0d618eac7f01/volumes" Sep 10 17:21:05.081734 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.081688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:05.183485 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.183457 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79d7459bb-4phmh"] Sep 10 17:21:05.183837 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.183824 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca25a0d9-371a-4f44-8d80-0d618eac7f01" containerName="authorino" Sep 10 17:21:05.183895 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.183842 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca25a0d9-371a-4f44-8d80-0d618eac7f01" containerName="authorino" Sep 10 17:21:05.183928 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.183916 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca25a0d9-371a-4f44-8d80-0d618eac7f01" containerName="authorino" Sep 10 17:21:05.191875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.191854 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.197434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.197412 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 17:21:05.197565 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.197412 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8v7fk\"" Sep 10 17:21:05.197565 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.197415 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 17:21:05.197745 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.197733 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 17:21:05.209678 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.209656 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79d7459bb-4phmh"] Sep 10 17:21:05.277635 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.277596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:05.301791 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.301761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd9xm\" (UniqueName: \"kubernetes.io/projected/b29b1e66-4442-4e14-8936-744cfd57de94-kube-api-access-pd9xm\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.301979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.301847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-webhook-cert\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.301979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.301939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-metrics-certs\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.403171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.403077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd9xm\" (UniqueName: \"kubernetes.io/projected/b29b1e66-4442-4e14-8936-744cfd57de94-kube-api-access-pd9xm\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.403351 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.403234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-webhook-cert\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.403351 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.403302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-metrics-certs\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.405632 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.405608 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-metrics-certs\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.405722 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.405663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-webhook-cert\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.418110 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.418081 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd9xm\" (UniqueName: \"kubernetes.io/projected/b29b1e66-4442-4e14-8936-744cfd57de94-kube-api-access-pd9xm\") pod \"maas-controller-79d7459bb-4phmh\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.484309 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.484273 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6768fddc-l8k2k"] Sep 10 17:21:05.487943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.487926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.496458 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.496437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:05.501773 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.501757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:05.517860 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.517818 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6768fddc-l8k2k"] Sep 10 17:21:05.605437 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.605401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-webhook-cert\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.605574 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.605478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx458\" (UniqueName: \"kubernetes.io/projected/d78a65f8-e841-4928-a7d5-3ac276472c77-kube-api-access-mx458\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.605574 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.605517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-metrics-certs\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.706350 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.706269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-webhook-cert\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.706350 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.706311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx458\" (UniqueName: \"kubernetes.io/projected/d78a65f8-e841-4928-a7d5-3ac276472c77-kube-api-access-mx458\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.706350 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.706344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-metrics-certs\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.708730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.708704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-metrics-certs\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.708831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.708745 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-webhook-cert\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.724541 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.724518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx458\" (UniqueName: \"kubernetes.io/projected/d78a65f8-e841-4928-a7d5-3ac276472c77-kube-api-access-mx458\") pod \"maas-controller-b6768fddc-l8k2k\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.798731 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.798699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:05.823129 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:05.823107 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79d7459bb-4phmh"] Sep 10 17:21:05.824702 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:21:05.824654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29b1e66_4442_4e14_8936_744cfd57de94.slice/crio-8a80b17227697da5fc1eb3674b82ca3dd8af6395706c1cbd68ef518cd92ad147 WatchSource:0}: Error finding container 8a80b17227697da5fc1eb3674b82ca3dd8af6395706c1cbd68ef518cd92ad147: Status 404 returned error can't find the container with id 8a80b17227697da5fc1eb3674b82ca3dd8af6395706c1cbd68ef518cd92ad147 Sep 10 17:21:06.089135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:06.089106 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6768fddc-l8k2k"] Sep 10 17:21:06.093856 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:21:06.093824 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78a65f8_e841_4928_a7d5_3ac276472c77.slice/crio-18e3b7ae85aaa8bdb7a70e3eafb8ded04a17cdb9a5ee59ae27b4b24e06c56839 WatchSource:0}: Error finding container 18e3b7ae85aaa8bdb7a70e3eafb8ded04a17cdb9a5ee59ae27b4b24e06c56839: Status 404 returned error can't find the container with id 18e3b7ae85aaa8bdb7a70e3eafb8ded04a17cdb9a5ee59ae27b4b24e06c56839 Sep 10 17:21:06.239381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:06.239345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-l8k2k" event={"ID":"d78a65f8-e841-4928-a7d5-3ac276472c77","Type":"ContainerStarted","Data":"18e3b7ae85aaa8bdb7a70e3eafb8ded04a17cdb9a5ee59ae27b4b24e06c56839"} Sep 10 17:21:06.240554 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:06.240529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-4phmh" event={"ID":"b29b1e66-4442-4e14-8936-744cfd57de94","Type":"ContainerStarted","Data":"8a80b17227697da5fc1eb3674b82ca3dd8af6395706c1cbd68ef518cd92ad147"} Sep 10 17:21:08.781733 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:08.781647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:09.080597 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:09.080567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:09.261905 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:09.261845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-l8k2k" event={"ID":"d78a65f8-e841-4928-a7d5-3ac276472c77","Type":"ContainerStarted","Data":"78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe"} Sep 10 17:21:09.262087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:09.262020 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:09.263494 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:09.263469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-4phmh" event={"ID":"b29b1e66-4442-4e14-8936-744cfd57de94","Type":"ContainerStarted","Data":"10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056"} Sep 10 17:21:09.263622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:09.263605 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:09.301289 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:09.301244 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6768fddc-l8k2k" podStartSLOduration=1.9406736850000001 podStartE2EDuration="4.301231522s" totalImagesPullingTime="2.360557837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:21:09.297592655 +0000 UTC m=+575.278672653" watchObservedRunningTime="2026-09-10 17:21:09.301231522 +0000 UTC m=+575.282311520" Sep 10 17:21:10.255108 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:10.255074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:20.270302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.270270 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:20.270705 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.270338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:21:20.296922 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.296860 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79d7459bb-4phmh" podStartSLOduration=12.67744527 podStartE2EDuration="15.296845191s" totalImagesPullingTime="2.619399921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:21:09.355533109 +0000 UTC m=+575.336613106" watchObservedRunningTime="2026-09-10 17:21:20.296845191 +0000 UTC m=+586.277925188" Sep 10 17:21:20.362833 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.362802 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79d7459bb-4phmh"] Sep 10 17:21:20.363036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.363009 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79d7459bb-4phmh" podUID="b29b1e66-4442-4e14-8936-744cfd57de94" containerName="manager" containerID="cri-o://10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056" gracePeriod=30 Sep 10 17:21:20.614302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.614279 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:20.764977 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.764944 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd9xm\" (UniqueName: \"kubernetes.io/projected/b29b1e66-4442-4e14-8936-744cfd57de94-kube-api-access-pd9xm\") pod \"b29b1e66-4442-4e14-8936-744cfd57de94\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " Sep 10 17:21:20.765143 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.765050 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-webhook-cert\") pod \"b29b1e66-4442-4e14-8936-744cfd57de94\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " Sep 10 17:21:20.765143 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.765082 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-metrics-certs\") pod \"b29b1e66-4442-4e14-8936-744cfd57de94\" (UID: \"b29b1e66-4442-4e14-8936-744cfd57de94\") " Sep 10 17:21:20.767233 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.767205 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29b1e66-4442-4e14-8936-744cfd57de94-kube-api-access-pd9xm" (OuterVolumeSpecName: "kube-api-access-pd9xm") pod "b29b1e66-4442-4e14-8936-744cfd57de94" (UID: "b29b1e66-4442-4e14-8936-744cfd57de94"). InnerVolumeSpecName "kube-api-access-pd9xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:21:20.767347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.767278 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b29b1e66-4442-4e14-8936-744cfd57de94" (UID: "b29b1e66-4442-4e14-8936-744cfd57de94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:21:20.767347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.767289 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b29b1e66-4442-4e14-8936-744cfd57de94" (UID: "b29b1e66-4442-4e14-8936-744cfd57de94"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:21:20.866058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.866027 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-webhook-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:21:20.866058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.866052 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29b1e66-4442-4e14-8936-744cfd57de94-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:21:20.866058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:20.866063 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd9xm\" (UniqueName: \"kubernetes.io/projected/b29b1e66-4442-4e14-8936-744cfd57de94-kube-api-access-pd9xm\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:21:21.329676 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.329639 2576 generic.go:356] "Generic (PLEG): container finished" podID="b29b1e66-4442-4e14-8936-744cfd57de94" containerID="10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056" exitCode=0 Sep 10 17:21:21.330103 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.329687 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7459bb-4phmh" Sep 10 17:21:21.330103 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.329699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-4phmh" event={"ID":"b29b1e66-4442-4e14-8936-744cfd57de94","Type":"ContainerDied","Data":"10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056"} Sep 10 17:21:21.330103 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.329754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7459bb-4phmh" event={"ID":"b29b1e66-4442-4e14-8936-744cfd57de94","Type":"ContainerDied","Data":"8a80b17227697da5fc1eb3674b82ca3dd8af6395706c1cbd68ef518cd92ad147"} Sep 10 17:21:21.330103 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.329779 2576 scope.go:117] "RemoveContainer" containerID="10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056" Sep 10 17:21:21.339167 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.339151 2576 scope.go:117] "RemoveContainer" containerID="10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056" Sep 10 17:21:21.339411 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:21:21.339391 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056\": container with ID starting with 10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056 not found: ID does not exist" containerID="10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056" Sep 10 17:21:21.339475 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.339423 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056"} err="failed to get container status \"10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056\": rpc error: code = NotFound desc = could not find container \"10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056\": container with ID starting with 10b2ba200270b9627157a6acecf2b49a4b43bd9061707d174bd4c8d38738e056 not found: ID does not exist" Sep 10 17:21:21.355188 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.355167 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79d7459bb-4phmh"] Sep 10 17:21:21.363318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:21.363294 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79d7459bb-4phmh"] Sep 10 17:21:22.486380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:22.486352 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b29b1e66-4442-4e14-8936-744cfd57de94" path="/var/lib/kubelet/pods/b29b1e66-4442-4e14-8936-744cfd57de94/volumes" Sep 10 17:21:34.447360 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:34.447332 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:21:34.448666 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:34.448643 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:21:41.255778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:41.255726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:41.449471 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:41.449432 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:42.686108 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.686059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:42.883177 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.883131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:42.980667 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.980585 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76"] Sep 10 17:21:42.980995 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.980982 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29b1e66-4442-4e14-8936-744cfd57de94" containerName="manager" Sep 10 17:21:42.981047 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.980997 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29b1e66-4442-4e14-8936-744cfd57de94" containerName="manager" Sep 10 17:21:42.981083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.981059 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b29b1e66-4442-4e14-8936-744cfd57de94" containerName="manager" Sep 10 17:21:42.987526 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.987508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:42.991523 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.991503 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n7zfj\"" Sep 10 17:21:42.991642 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.991523 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 17:21:42.995196 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:42.995180 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 17:21:43.030254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.030220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-metrics-certs\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.030381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.030258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txttf\" (UniqueName: \"kubernetes.io/projected/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-kube-api-access-txttf\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.030381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.030320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-openshift-service-ca\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.030381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.030373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-maas-api-tls\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.036061 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.036035 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76"] Sep 10 17:21:43.131662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.131631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-maas-api-tls\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.131869 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.131695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-metrics-certs\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.131869 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.131726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txttf\" (UniqueName: \"kubernetes.io/projected/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-kube-api-access-txttf\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.131869 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.131777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-openshift-service-ca\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.132451 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.132428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-openshift-service-ca\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.134172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.134149 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-maas-api-tls\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.134261 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.134237 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-metrics-certs\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.156211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.156181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txttf\" (UniqueName: \"kubernetes.io/projected/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-kube-api-access-txttf\") pod \"maas-api-ff9f6d487-pgq76\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.176498 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.176467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:43.297439 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.297361 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:43.342943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.342904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:43.601760 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.601731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76"] Sep 10 17:21:43.602155 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:21:43.602128 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6177e1_985f_4bfc_b224_e97ce0fc1e3a.slice/crio-3a4bbbe532fddd07288bc87343c1b5e8d94611e649bf3c290c7e4665a47fe836 WatchSource:0}: Error finding container 3a4bbbe532fddd07288bc87343c1b5e8d94611e649bf3c290c7e4665a47fe836: Status 404 returned error can't find the container with id 3a4bbbe532fddd07288bc87343c1b5e8d94611e649bf3c290c7e4665a47fe836 Sep 10 17:21:43.872939 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:43.872827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:44.151070 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:44.150977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:44.462266 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:44.462186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" event={"ID":"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a","Type":"ContainerStarted","Data":"3a4bbbe532fddd07288bc87343c1b5e8d94611e649bf3c290c7e4665a47fe836"} Sep 10 17:21:44.632602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:44.632571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:46.476201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:46.476164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" event={"ID":"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a","Type":"ContainerStarted","Data":"38c0ece50b383c8c6d97143c0f546d7a24a00013cfc2c28a4c0e5eaf10bd7236"} Sep 10 17:21:46.476601 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:46.476277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:46.513509 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:46.513439 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" podStartSLOduration=2.329890639 podStartE2EDuration="4.513423822s" totalImagesPullingTime="2.183533183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:21:46.512166683 +0000 UTC m=+612.493246673" watchObservedRunningTime="2026-09-10 17:21:46.513423822 +0000 UTC m=+612.494503822" Sep 10 17:21:52.489675 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:52.489646 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:21:53.734324 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:53.734294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:53.906994 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:53.906960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:21:54.526782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:54.526751 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a35ba9e-05d0-4a6a-90dc-03e621ea9e04" containerID="448c4339ae9734d418b1f50f04cbef4251a539f052729564d5bf0161eae93b33" exitCode=0 Sep 10 17:21:54.526956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:54.526822 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" event={"ID":"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04","Type":"ContainerDied","Data":"448c4339ae9734d418b1f50f04cbef4251a539f052729564d5bf0161eae93b33"} Sep 10 17:21:54.527178 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:54.527164 2576 scope.go:117] "RemoveContainer" containerID="448c4339ae9734d418b1f50f04cbef4251a539f052729564d5bf0161eae93b33" Sep 10 17:21:54.528127 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:54.528102 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:21:54.528188 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:54.528174 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:21:54.528224 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:54.528208 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:21:55.534274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:55.534237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" event={"ID":"4a35ba9e-05d0-4a6a-90dc-03e621ea9e04","Type":"ContainerStarted","Data":"b513d54fd40be9a099ca1efb12ca8003bdf9f4a96e1cf6261bbf0f9478ed7ff8"} Sep 10 17:21:55.566192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:55.566160 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:21:55.567304 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:55.567277 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gmhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:21:55.567415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:55.567332 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" podUID="4a35ba9e-05d0-4a6a-90dc-03e621ea9e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 17:21:56.566105 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:56.566071 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gmhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:21:56.566490 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:56.566120 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" podUID="4a35ba9e-05d0-4a6a-90dc-03e621ea9e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 17:21:57.566584 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:57.566550 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gmhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:21:57.567000 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:57.566606 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" podUID="4a35ba9e-05d0-4a6a-90dc-03e621ea9e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 17:21:58.566034 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:58.565995 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gmhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:21:58.566234 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:58.566052 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" podUID="4a35ba9e-05d0-4a6a-90dc-03e621ea9e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 17:21:59.565979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:59.565943 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gmhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:21:59.566442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:59.566000 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" podUID="4a35ba9e-05d0-4a6a-90dc-03e621ea9e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 17:21:59.997683 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:21:59.997643 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:22:00.566168 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:00.566126 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gmhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:22:00.566579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:00.566193 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" podUID="4a35ba9e-05d0-4a6a-90dc-03e621ea9e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 10 17:22:01.572574 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.572546 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:22:01.573609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.573584 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gmhd" Sep 10 17:22:01.926769 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.926694 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm"] Sep 10 17:22:01.930295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.930277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:01.935271 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.935253 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mn569\"" Sep 10 17:22:01.935446 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.935419 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 17:22:01.935683 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.935666 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 17:22:01.936024 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.936008 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 17:22:01.991744 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:01.991693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:02.004783 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.004757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.004940 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.004831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.005021 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.004954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8wk\" (UniqueName: \"kubernetes.io/projected/6bfefb18-2b34-43ef-9708-e75e150d1ff9-kube-api-access-gc8wk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.005021 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.004988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfefb18-2b34-43ef-9708-e75e150d1ff9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.005021 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.005017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.005175 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.005064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.035046 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.035014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm"] Sep 10 17:22:02.106434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8wk\" (UniqueName: \"kubernetes.io/projected/6bfefb18-2b34-43ef-9708-e75e150d1ff9-kube-api-access-gc8wk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.106434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfefb18-2b34-43ef-9708-e75e150d1ff9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.106686 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.106686 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.106686 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.106686 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.106943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.107016 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.107016 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.106978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.108780 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.108757 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bfefb18-2b34-43ef-9708-e75e150d1ff9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.109015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.108998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bfefb18-2b34-43ef-9708-e75e150d1ff9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.117501 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.117470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8wk\" (UniqueName: \"kubernetes.io/projected/6bfefb18-2b34-43ef-9708-e75e150d1ff9-kube-api-access-gc8wk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm\" (UID: \"6bfefb18-2b34-43ef-9708-e75e150d1ff9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.240688 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.240600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:02.526196 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.526166 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm"] Sep 10 17:22:02.528066 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:22:02.528038 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfefb18_2b34_43ef_9708_e75e150d1ff9.slice/crio-fdaacc9526c41d0dc749fc0b4cfc8a42a91004312622e240161fe5f3492be152 WatchSource:0}: Error finding container fdaacc9526c41d0dc749fc0b4cfc8a42a91004312622e240161fe5f3492be152: Status 404 returned error can't find the container with id fdaacc9526c41d0dc749fc0b4cfc8a42a91004312622e240161fe5f3492be152 Sep 10 17:22:02.530177 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.530152 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:22:02.575205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:02.575165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" event={"ID":"6bfefb18-2b34-43ef-9708-e75e150d1ff9","Type":"ContainerStarted","Data":"fdaacc9526c41d0dc749fc0b4cfc8a42a91004312622e240161fe5f3492be152"} Sep 10 17:22:04.587291 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:04.587258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" event={"ID":"6bfefb18-2b34-43ef-9708-e75e150d1ff9","Type":"ContainerStarted","Data":"17b1129c073db10e935e7fd709b8012840fb2bab009366deee9771f1f34f7752"} Sep 10 17:22:04.619362 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:04.619311 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" podStartSLOduration=2.425299645 podStartE2EDuration="3.619295818s" totalImagesPullingTime="1.193996173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:04.615921634 +0000 UTC m=+630.597001629" watchObservedRunningTime="2026-09-10 17:22:04.619295818 +0000 UTC m=+630.600375840" Sep 10 17:22:12.060123 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:12.060082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:12.241249 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:12.241215 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:12.241429 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:12.241304 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:12.253842 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:12.253807 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:12.642459 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:12.642429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm" Sep 10 17:22:13.139606 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:13.139571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:13.414519 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:13.414437 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:18.342637 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:18.342600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:25.271766 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:25.271730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:25.379992 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:25.379953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:25.617370 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:25.617335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:37.771008 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.770973 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp"] Sep 10 17:22:37.773644 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.773622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.776963 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.776944 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 17:22:37.803365 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.803339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp"] Sep 10 17:22:37.824020 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.823981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.824134 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.824058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.824134 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.824089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55bbd2be-f242-4686-9a55-e177df48f59c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.824134 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.824110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqbp\" (UniqueName: \"kubernetes.io/projected/55bbd2be-f242-4686-9a55-e177df48f59c-kube-api-access-7kqbp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.824244 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.824183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.824279 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.824246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.845532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.845503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:37.924917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.924873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55bbd2be-f242-4686-9a55-e177df48f59c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.924932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqbp\" (UniqueName: \"kubernetes.io/projected/55bbd2be-f242-4686-9a55-e177df48f59c-kube-api-access-7kqbp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.924956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.925009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.925031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.925102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925423 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.925402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.925441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.925500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.925480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.927378 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.927359 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55bbd2be-f242-4686-9a55-e177df48f59c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.927378 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.927370 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55bbd2be-f242-4686-9a55-e177df48f59c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:37.935322 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:37.935302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqbp\" (UniqueName: \"kubernetes.io/projected/55bbd2be-f242-4686-9a55-e177df48f59c-kube-api-access-7kqbp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d2rnp\" (UID: \"55bbd2be-f242-4686-9a55-e177df48f59c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:38.083539 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:38.083506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:38.384535 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:38.384511 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp"] Sep 10 17:22:38.386611 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:22:38.386586 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bbd2be_f242_4686_9a55_e177df48f59c.slice/crio-7824ce53527537f60cc041f3a43bc6b59025731f4a78efd2c77e2cb9743a1d4a WatchSource:0}: Error finding container 7824ce53527537f60cc041f3a43bc6b59025731f4a78efd2c77e2cb9743a1d4a: Status 404 returned error can't find the container with id 7824ce53527537f60cc041f3a43bc6b59025731f4a78efd2c77e2cb9743a1d4a Sep 10 17:22:38.779570 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:38.779482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" event={"ID":"55bbd2be-f242-4686-9a55-e177df48f59c","Type":"ContainerStarted","Data":"8703d1e995881dc6fdfb630f86b6787815be6fb0a1155f68514b4d1bc9674039"} Sep 10 17:22:38.779570 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:38.779543 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" event={"ID":"55bbd2be-f242-4686-9a55-e177df48f59c","Type":"ContainerStarted","Data":"7824ce53527537f60cc041f3a43bc6b59025731f4a78efd2c77e2cb9743a1d4a"} Sep 10 17:22:38.813698 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:38.813647 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" podStartSLOduration=1.6412429560000001 podStartE2EDuration="1.813632848s" totalImagesPullingTime="172.389892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:38.80964085 +0000 UTC m=+664.790720847" watchObservedRunningTime="2026-09-10 17:22:38.813632848 +0000 UTC m=+664.794712847" Sep 10 17:22:46.699386 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:46.699352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:48.084306 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:48.084273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:48.084718 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:48.084441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:48.097162 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:48.097134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:48.847677 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:48.847644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d2rnp" Sep 10 17:22:56.129430 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.129394 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz"] Sep 10 17:22:56.132559 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.132523 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.141350 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.141324 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 17:22:56.156284 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.156256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz"] Sep 10 17:22:56.188503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.188457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:22:56.194551 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.194527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.194682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.194563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.194682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.194588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.194682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.194649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.194801 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.194723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shcgt\" (UniqueName: \"kubernetes.io/projected/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-kube-api-access-shcgt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.194801 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.194747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296069 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296239 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296239 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296239 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296377 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shcgt\" (UniqueName: \"kubernetes.io/projected/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-kube-api-access-shcgt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296420 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296557 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296705 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.296769 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.296750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.298306 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.298287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.298603 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.298586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.307372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.307343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shcgt\" (UniqueName: \"kubernetes.io/projected/4633c4e7-0c65-42db-b36d-fc7b0dfee9f1-kube-api-access-shcgt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-p8hxz\" (UID: \"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.444188 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.444102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:22:56.725431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.725401 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz"] Sep 10 17:22:56.726618 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:22:56.726589 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4633c4e7_0c65_42db_b36d_fc7b0dfee9f1.slice/crio-ba2a06f1ab1a9973a62339e807e30e4a9f94644dea0df88408bdf2ec6f497ea9 WatchSource:0}: Error finding container ba2a06f1ab1a9973a62339e807e30e4a9f94644dea0df88408bdf2ec6f497ea9: Status 404 returned error can't find the container with id ba2a06f1ab1a9973a62339e807e30e4a9f94644dea0df88408bdf2ec6f497ea9 Sep 10 17:22:56.879073 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:56.879039 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" event={"ID":"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1","Type":"ContainerStarted","Data":"ba2a06f1ab1a9973a62339e807e30e4a9f94644dea0df88408bdf2ec6f497ea9"} Sep 10 17:22:57.888708 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:57.888666 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" event={"ID":"4633c4e7-0c65-42db-b36d-fc7b0dfee9f1","Type":"ContainerStarted","Data":"b3837973d08cc3af3ef75a91a3c843f378002a8fb89c19c1828fc0e62a21924c"} Sep 10 17:22:57.917506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:22:57.917452 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" podStartSLOduration=1.6682561009999999 podStartE2EDuration="1.917434781s" totalImagesPullingTime="249.17868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:57.915287112 +0000 UTC m=+683.896367109" watchObservedRunningTime="2026-09-10 17:22:57.917434781 +0000 UTC m=+683.898514778" Sep 10 17:23:01.897773 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:01.897735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:23:02.020556 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:02.020524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:23:06.444455 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:06.444416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:23:06.444905 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:06.444503 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:23:06.456960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:06.456934 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:23:06.949974 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:06.949947 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-p8hxz" Sep 10 17:23:10.425077 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:10.425043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:23:30.130755 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:23:30.130672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:00.345648 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:00.345606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:00.422838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:00.422799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:01.683426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:01.683391 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f4c699d65-4f4kx"] Sep 10 17:24:01.684008 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:01.683573 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" podUID="ce9c3f12-15a7-4362-adb2-b8eb423d6d33" containerName="authorino" containerID="cri-o://af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84" gracePeriod=30 Sep 10 17:24:01.946170 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:01.946110 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:24:02.093602 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.093559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-tls-cert\") pod \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " Sep 10 17:24:02.093800 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.093677 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgpq5\" (UniqueName: \"kubernetes.io/projected/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-kube-api-access-rgpq5\") pod \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\" (UID: \"ce9c3f12-15a7-4362-adb2-b8eb423d6d33\") " Sep 10 17:24:02.095722 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.095689 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-kube-api-access-rgpq5" (OuterVolumeSpecName: "kube-api-access-rgpq5") pod "ce9c3f12-15a7-4362-adb2-b8eb423d6d33" (UID: "ce9c3f12-15a7-4362-adb2-b8eb423d6d33"). InnerVolumeSpecName "kube-api-access-rgpq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:24:02.104409 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.104384 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ce9c3f12-15a7-4362-adb2-b8eb423d6d33" (UID: "ce9c3f12-15a7-4362-adb2-b8eb423d6d33"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:24:02.194795 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.194760 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgpq5\" (UniqueName: \"kubernetes.io/projected/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-kube-api-access-rgpq5\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:24:02.194795 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.194789 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce9c3f12-15a7-4362-adb2-b8eb423d6d33-tls-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:24:02.235792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.235712 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce9c3f12-15a7-4362-adb2-b8eb423d6d33" containerID="af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84" exitCode=0 Sep 10 17:24:02.235792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.235755 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" Sep 10 17:24:02.236036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.235793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" event={"ID":"ce9c3f12-15a7-4362-adb2-b8eb423d6d33","Type":"ContainerDied","Data":"af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84"} Sep 10 17:24:02.236036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.235841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4c699d65-4f4kx" event={"ID":"ce9c3f12-15a7-4362-adb2-b8eb423d6d33","Type":"ContainerDied","Data":"dd59d6fb4eaca632cabaf189ee1ecbd1ec4740c296874933a184431a49327c65"} Sep 10 17:24:02.236036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.235863 2576 scope.go:117] "RemoveContainer" containerID="af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84" Sep 10 17:24:02.246213 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.246191 2576 scope.go:117] "RemoveContainer" containerID="af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84" Sep 10 17:24:02.246565 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:24:02.246540 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84\": container with ID starting with af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84 not found: ID does not exist" containerID="af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84" Sep 10 17:24:02.246629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.246575 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84"} err="failed to get container status \"af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84\": rpc error: code = NotFound desc = could not find container \"af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84\": container with ID starting with af8596d723317a2e3813b04236196ef1dfd7946e6b57d4ca45a1e56f82a81c84 not found: ID does not exist" Sep 10 17:24:02.271302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.271270 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f4c699d65-4f4kx"] Sep 10 17:24:02.277793 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.277767 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f4c699d65-4f4kx"] Sep 10 17:24:02.314919 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.314853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:02.465175 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.465135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:02.487862 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:02.487790 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9c3f12-15a7-4362-adb2-b8eb423d6d33" path="/var/lib/kubelet/pods/ce9c3f12-15a7-4362-adb2-b8eb423d6d33/volumes" Sep 10 17:24:03.503876 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:03.503840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:03.639002 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:03.638968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:14.351371 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:14.351326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:25.431165 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:25.431121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:25.654782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:25.654741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:25.910526 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:25.910490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:26.094317 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.094282 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r"] Sep 10 17:24:26.094864 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.094845 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9c3f12-15a7-4362-adb2-b8eb423d6d33" containerName="authorino" Sep 10 17:24:26.094962 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.094867 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9c3f12-15a7-4362-adb2-b8eb423d6d33" containerName="authorino" Sep 10 17:24:26.095057 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.095044 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9c3f12-15a7-4362-adb2-b8eb423d6d33" containerName="authorino" Sep 10 17:24:26.097808 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.097793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.112732 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.112702 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dc5a37-openshift-default-dockercfg-kptst\"" Sep 10 17:24:26.157262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.157225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r"] Sep 10 17:24:26.162623 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.162537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:26.195625 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-data\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.195825 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-envoy\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.195825 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-certs\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.195825 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzps\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-kube-api-access-kpzps\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.195825 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-token\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.195825 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istiod-ca-cert\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.196081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-podinfo\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.196081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-socket\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.196081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.195972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-credential-socket\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.296625 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-data\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.296826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-envoy\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.296826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-certs\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.296826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzps\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-kube-api-access-kpzps\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.296826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-token\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.296826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istiod-ca-cert\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-podinfo\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-socket\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.296924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-credential-socket\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.297068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-data\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.297265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-credential-socket\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.297352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-socket\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297678 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.297651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-certs\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.297790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.297680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istiod-ca-cert\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.299496 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.299451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-envoy\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.299941 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.299909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-podinfo\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.351253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.351222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-token\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.354080 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.354054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzps\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-kube-api-access-kpzps\") pod \"e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.409200 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.409166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:26.433714 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.433682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:26.709971 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.708590 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r"] Sep 10 17:24:26.713486 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:24:26.713452 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5633e8_f77a_4313_a1f8_9fd99bf8b478.slice/crio-e5e886abcb5ba7f29436b5a21a7db98b4dbd1572fa9d0008ef097bea0f01c12a WatchSource:0}: Error finding container e5e886abcb5ba7f29436b5a21a7db98b4dbd1572fa9d0008ef097bea0f01c12a: Status 404 returned error can't find the container with id e5e886abcb5ba7f29436b5a21a7db98b4dbd1572fa9d0008ef097bea0f01c12a Sep 10 17:24:26.715576 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.715542 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:24:26.715669 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.715606 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:24:26.715669 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:26.715633 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:24:27.259108 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.259073 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:27.378366 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.378331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" event={"ID":"1a5633e8-f77a-4313-a1f8-9fd99bf8b478","Type":"ContainerStarted","Data":"a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a"} Sep 10 17:24:27.378521 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.378390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" event={"ID":"1a5633e8-f77a-4313-a1f8-9fd99bf8b478","Type":"ContainerStarted","Data":"e5e886abcb5ba7f29436b5a21a7db98b4dbd1572fa9d0008ef097bea0f01c12a"} Sep 10 17:24:27.409507 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.409477 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:27.414498 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.414469 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:27.422500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.422456 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" podStartSLOduration=1.422442059 podStartE2EDuration="1.422442059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:27.420045552 +0000 UTC m=+773.401125550" watchObservedRunningTime="2026-09-10 17:24:27.422442059 +0000 UTC m=+773.403522057" Sep 10 17:24:27.463400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.463366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:27.776564 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:27.776522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:28.384816 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:28.384785 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:28.385699 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:28.385679 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:29.721140 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:29.721107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:29.860385 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:29.860349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:31.125968 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.125567 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd"] Sep 10 17:24:31.136963 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.136934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.140657 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.140630 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dc5a37-serving-cert\"" Sep 10 17:24:31.153117 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.153087 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd"] Sep 10 17:24:31.216115 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.216080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:31.250401 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.250360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.250401 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.250395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-metrics-certs\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.250655 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.250436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kz2d\" (UniqueName: \"kubernetes.io/projected/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-kube-api-access-4kz2d\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.250725 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.250660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.333074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.333040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:31.351283 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.351244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kz2d\" (UniqueName: \"kubernetes.io/projected/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-kube-api-access-4kz2d\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.351440 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.351307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.351440 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.351363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.351440 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.351379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-metrics-certs\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.351604 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:24:31.351493 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-serving-cert: secret "maas-api-e2e-inf-a-dc5a37-serving-cert" not found Sep 10 17:24:31.351604 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:24:31.351586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls podName:8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e nodeName:}" failed. No retries permitted until 2026-09-10 17:24:31.85156221 +0000 UTC m=+777.832642204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls") pod "maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" (UID: "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e") : secret "maas-api-e2e-inf-a-dc5a37-serving-cert" not found Sep 10 17:24:31.352045 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.352024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.353766 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.353748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-metrics-certs\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.370573 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.370540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kz2d\" (UniqueName: \"kubernetes.io/projected/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-kube-api-access-4kz2d\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.745464 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.745423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:31.857058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.857025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:31.859527 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:31.859492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:32.050845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:32.050763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:32.435143 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:32.435109 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd"] Sep 10 17:24:32.437047 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:24:32.437011 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3cf0b5_c7d6_4bb3_a1d9_0738fd5dbb4e.slice/crio-b81dc9c297acfd1ec997dd3402d711c08499fef727a253c27a56ae7ab205f71b WatchSource:0}: Error finding container b81dc9c297acfd1ec997dd3402d711c08499fef727a253c27a56ae7ab205f71b: Status 404 returned error can't find the container with id b81dc9c297acfd1ec997dd3402d711c08499fef727a253c27a56ae7ab205f71b Sep 10 17:24:32.503712 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:32.503671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:32.653046 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:32.653012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:33.416698 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:33.416656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" event={"ID":"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e","Type":"ContainerStarted","Data":"135fcd64b4099fcb411623fe63db3923e30c23602c5d7daaaaed7362c3103916"} Sep 10 17:24:33.416982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:33.416711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" event={"ID":"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e","Type":"ContainerStarted","Data":"b81dc9c297acfd1ec997dd3402d711c08499fef727a253c27a56ae7ab205f71b"} Sep 10 17:24:33.416982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:33.416835 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:33.457107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:33.457051 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" podStartSLOduration=1.960634481 podStartE2EDuration="2.457030031s" totalImagesPullingTime="496.39555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:33.452763325 +0000 UTC m=+779.433843334" watchObservedRunningTime="2026-09-10 17:24:33.457030031 +0000 UTC m=+779.438110030" Sep 10 17:24:36.249658 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:36.249615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:36.399147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:36.399109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:36.650810 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:36.650774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:36.856532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:36.856498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:37.075402 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:37.075366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:37.264984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:37.264948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:39.428131 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:39.428104 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:24:41.962108 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:41.962067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:42.155137 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:42.155093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:42.516718 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:42.516684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:42.650598 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:42.650562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:43.196101 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.196056 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc"] Sep 10 17:24:43.200702 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.200668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.204312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.204288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-99ff0092-serving-cert\"" Sep 10 17:24:43.218818 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.218785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc"] Sep 10 17:24:43.274448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.274415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqcv\" (UniqueName: \"kubernetes.io/projected/bcc29a89-0d09-41dc-8c38-f8c272fd7689-kube-api-access-dfqcv\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.274630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.274470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.274630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.274586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-metrics-certs\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.274630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.274617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc29a89-0d09-41dc-8c38-f8c272fd7689-openshift-service-ca\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.315898 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.315853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:43.375850 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.375813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqcv\" (UniqueName: \"kubernetes.io/projected/bcc29a89-0d09-41dc-8c38-f8c272fd7689-kube-api-access-dfqcv\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.376083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.375912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.376083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.376007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-metrics-certs\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.376083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.376035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc29a89-0d09-41dc-8c38-f8c272fd7689-openshift-service-ca\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.376264 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:24:43.376148 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-serving-cert: secret "maas-api-e2e-ait-99ff0092-serving-cert" not found Sep 10 17:24:43.376264 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:24:43.376240 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls podName:bcc29a89-0d09-41dc-8c38-f8c272fd7689 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:43.876217955 +0000 UTC m=+789.857297935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls") pod "maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" (UID: "bcc29a89-0d09-41dc-8c38-f8c272fd7689") : secret "maas-api-e2e-ait-99ff0092-serving-cert" not found Sep 10 17:24:43.376895 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.376853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc29a89-0d09-41dc-8c38-f8c272fd7689-openshift-service-ca\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.379600 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.379575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-metrics-certs\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.396337 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.396300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqcv\" (UniqueName: \"kubernetes.io/projected/bcc29a89-0d09-41dc-8c38-f8c272fd7689-kube-api-access-dfqcv\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.433305 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.433267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:43.672191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.672135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:43.881327 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.881293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:43.883732 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:43.883706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls\") pod \"maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:44.119114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.119067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:44.156133 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.156094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:44.258738 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.258702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:44.412043 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.412011 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc"] Sep 10 17:24:44.413657 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:24:44.413629 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc29a89_0d09_41dc_8c38_f8c272fd7689.slice/crio-c7209413639705457c4687bef3602d2e56a7216e43e3fe5d10ffbb74a1874bc0 WatchSource:0}: Error finding container c7209413639705457c4687bef3602d2e56a7216e43e3fe5d10ffbb74a1874bc0: Status 404 returned error can't find the container with id c7209413639705457c4687bef3602d2e56a7216e43e3fe5d10ffbb74a1874bc0 Sep 10 17:24:44.478556 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.478519 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" event={"ID":"bcc29a89-0d09-41dc-8c38-f8c272fd7689","Type":"ContainerStarted","Data":"c7209413639705457c4687bef3602d2e56a7216e43e3fe5d10ffbb74a1874bc0"} Sep 10 17:24:44.762467 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.762440 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv"] Sep 10 17:24:44.767187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.767160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.790107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.790048 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv"] Sep 10 17:24:44.790555 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.790465 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-889898-serving-cert\"" Sep 10 17:24:44.857570 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.857531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:44.893420 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.893387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742vr\" (UniqueName: \"kubernetes.io/projected/4afd73a8-5881-4e80-9261-356dbd2982d1-kube-api-access-742vr\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.893627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.893514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-metrics-certs\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.893627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.893574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afd73a8-5881-4e80-9261-356dbd2982d1-openshift-service-ca\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.893627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.893616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.971818 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.971763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:44.995008 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.994896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742vr\" (UniqueName: \"kubernetes.io/projected/4afd73a8-5881-4e80-9261-356dbd2982d1-kube-api-access-742vr\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.995224 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.995053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-metrics-certs\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.995224 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.995093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afd73a8-5881-4e80-9261-356dbd2982d1-openshift-service-ca\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.995224 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.995136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.995412 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:24:44.995282 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-889898-serving-cert: secret "maas-api-e2e-autores-889898-serving-cert" not found Sep 10 17:24:44.995412 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:24:44.995361 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls podName:4afd73a8-5881-4e80-9261-356dbd2982d1 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:45.495338558 +0000 UTC m=+791.476418534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls") pod "maas-api-e2e-autores-889898-dc67cb7f-q6xrv" (UID: "4afd73a8-5881-4e80-9261-356dbd2982d1") : secret "maas-api-e2e-autores-889898-serving-cert" not found Sep 10 17:24:44.995947 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.995908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afd73a8-5881-4e80-9261-356dbd2982d1-openshift-service-ca\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:44.998420 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:44.998386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-metrics-certs\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:45.026789 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.026752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742vr\" (UniqueName: \"kubernetes.io/projected/4afd73a8-5881-4e80-9261-356dbd2982d1-kube-api-access-742vr\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:45.095913 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.095850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:45.230999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.230964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:45.335241 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.335204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:45.487294 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.487243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" event={"ID":"bcc29a89-0d09-41dc-8c38-f8c272fd7689","Type":"ContainerStarted","Data":"a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16"} Sep 10 17:24:45.487485 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.487323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:45.500770 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.500737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:45.503713 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.503678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls\") pod \"maas-api-e2e-autores-889898-dc67cb7f-q6xrv\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:45.520492 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.520435 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" podStartSLOduration=2.202875959 podStartE2EDuration="2.52041749s" totalImagesPullingTime="317.541531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:45.517954165 +0000 UTC m=+791.499034167" watchObservedRunningTime="2026-09-10 17:24:45.52041749 +0000 UTC m=+791.501497488" Sep 10 17:24:45.685803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.685700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:45.704102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.704065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:45.871235 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:45.871198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:46.005082 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:46.005047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv"] Sep 10 17:24:46.005577 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:24:46.005548 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afd73a8_5881_4e80_9261_356dbd2982d1.slice/crio-52f2fcd48fffe51c4f8a34ac1e9c907414101b1eb2748d0eb1110d9b88fe2059 WatchSource:0}: Error finding container 52f2fcd48fffe51c4f8a34ac1e9c907414101b1eb2748d0eb1110d9b88fe2059: Status 404 returned error can't find the container with id 52f2fcd48fffe51c4f8a34ac1e9c907414101b1eb2748d0eb1110d9b88fe2059 Sep 10 17:24:46.495782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:46.495741 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" event={"ID":"4afd73a8-5881-4e80-9261-356dbd2982d1","Type":"ContainerStarted","Data":"1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf"} Sep 10 17:24:46.496262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:46.495801 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" event={"ID":"4afd73a8-5881-4e80-9261-356dbd2982d1","Type":"ContainerStarted","Data":"52f2fcd48fffe51c4f8a34ac1e9c907414101b1eb2748d0eb1110d9b88fe2059"} Sep 10 17:24:46.496262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:46.496084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:46.521774 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:46.521723 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" podStartSLOduration=2.184520474 podStartE2EDuration="2.521705845s" totalImagesPullingTime="337.185371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:46.518014072 +0000 UTC m=+792.499094074" watchObservedRunningTime="2026-09-10 17:24:46.521705845 +0000 UTC m=+792.502785842" Sep 10 17:24:46.541361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:46.541326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:47.664788 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:47.664746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:48.024892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:48.024784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:48.507984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:48.507955 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerID="a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a" exitCode=0 Sep 10 17:24:48.508169 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:48.508030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" event={"ID":"1a5633e8-f77a-4313-a1f8-9fd99bf8b478","Type":"ContainerDied","Data":"a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a"} Sep 10 17:24:48.508541 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:48.508523 2576 scope.go:117] "RemoveContainer" containerID="a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a" Sep 10 17:24:48.509618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:48.509590 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:24:48.509689 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:48.509676 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:24:48.509730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:48.509712 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:24:49.517579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:49.517542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" event={"ID":"1a5633e8-f77a-4313-a1f8-9fd99bf8b478","Type":"ContainerStarted","Data":"1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002"} Sep 10 17:24:50.409615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:50.409583 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:50.410198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:50.410165 2576 patch_prober.go:28] interesting pod/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:24:50.410311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:50.410241 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 17:24:50.471600 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:50.471565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:51.410700 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:51.410198 2576 patch_prober.go:28] interesting pod/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:24:51.410700 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:51.410285 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 17:24:51.492888 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:51.492836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:51.511166 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:51.511129 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:24:51.647779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:51.647741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:51.794618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:51.794510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:52.409805 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:52.409761 2576 patch_prober.go:28] interesting pod/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:24:52.410050 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:52.409844 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 17:24:52.511181 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:52.511137 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:24:53.369487 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:53.369449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:53.409837 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:53.409789 2576 patch_prober.go:28] interesting pod/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:24:53.410513 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:53.410479 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 17:24:54.410389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:54.410338 2576 patch_prober.go:28] interesting pod/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:24:54.410908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:54.410412 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 17:24:55.414192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:55.414159 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:55.414633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:55.414566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:55.415309 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:55.415292 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:24:58.792854 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:58.792774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:58.909149 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:58.909111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:59.011517 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:59.011480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:59.177379 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:59.177347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:59.660196 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:59.660119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:24:59.944475 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:24:59.944397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:00.321753 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:00.321716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:00.627597 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:00.627492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:03.482370 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:03.482328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:03.584615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:03.584581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:04.792720 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.792687 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86"] Sep 10 17:25:04.796595 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.796572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:04.800930 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.800908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-75bd9a-serving-cert\"" Sep 10 17:25:04.819482 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.819443 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86"] Sep 10 17:25:04.845470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.845436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:04.905064 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.905020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpt52\" (UniqueName: \"kubernetes.io/projected/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-kube-api-access-bpt52\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:04.905265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.905097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-metrics-certs\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:04.905265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.905129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:04.905361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.905263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:04.960728 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:04.960688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:05.005866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.005812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpt52\" (UniqueName: \"kubernetes.io/projected/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-kube-api-access-bpt52\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.006072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.005948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-metrics-certs\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.006072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.005982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.006148 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.006099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.006249 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:25:05.006232 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-serving-cert: secret "maas-api-e2e-inf-b-75bd9a-serving-cert" not found Sep 10 17:25:05.006317 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:25:05.006308 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls podName:b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe nodeName:}" failed. No retries permitted until 2026-09-10 17:25:05.506288586 +0000 UTC m=+811.487368575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls") pod "maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" (UID: "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe") : secret "maas-api-e2e-inf-b-75bd9a-serving-cert" not found Sep 10 17:25:05.006703 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.006683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.008484 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.008466 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-metrics-certs\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.019195 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.019165 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpt52\" (UniqueName: \"kubernetes.io/projected/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-kube-api-access-bpt52\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.219510 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.219473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:05.511399 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.511292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.513667 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.513640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls\") pod \"maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:05.566954 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.566914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:05.690633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.690589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:05.708973 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:05.708927 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:06.392814 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:25:06.392783 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cfcd8a_9e20_4be6_a942_8f3cc2a5eafe.slice/crio-9b2ba5ca6a6ee50cc5b0e32af03508a791ad069e3f2b27ed28698fdcdac131ab WatchSource:0}: Error finding container 9b2ba5ca6a6ee50cc5b0e32af03508a791ad069e3f2b27ed28698fdcdac131ab: Status 404 returned error can't find the container with id 9b2ba5ca6a6ee50cc5b0e32af03508a791ad069e3f2b27ed28698fdcdac131ab Sep 10 17:25:06.393984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:06.393951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86"] Sep 10 17:25:06.628250 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:06.628211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" event={"ID":"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe","Type":"ContainerStarted","Data":"9b2ba5ca6a6ee50cc5b0e32af03508a791ad069e3f2b27ed28698fdcdac131ab"} Sep 10 17:25:06.682964 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:06.682852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:07.635486 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:07.635452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" event={"ID":"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe","Type":"ContainerStarted","Data":"f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f"} Sep 10 17:25:07.635866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:07.635539 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:07.689372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:07.689324 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" podStartSLOduration=3.304428728 podStartE2EDuration="3.689305245s" totalImagesPullingTime="384.876517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:07.68641639 +0000 UTC m=+813.667496399" watchObservedRunningTime="2026-09-10 17:25:07.689305245 +0000 UTC m=+813.670385244" Sep 10 17:25:07.986901 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:07.986792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:10.078010 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:10.077965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:10.916816 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:10.916772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:11.088071 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:11.088029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:11.439236 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:11.439204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:12.663935 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:12.663858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:12.844123 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:12.844085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:13.647518 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:13.647481 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:25:13.754413 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:13.754382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:22.471596 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:22.471552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:24.125132 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.125087 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc"] Sep 10 17:25:24.125621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.125394 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" podUID="bcc29a89-0d09-41dc-8c38-f8c272fd7689" containerName="maas-api" containerID="cri-o://a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16" gracePeriod=30 Sep 10 17:25:24.159932 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.159836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:24.255094 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.255021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:24.340959 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.340910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:24.398269 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.398241 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:25:24.454074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.454040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:24.502616 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.502582 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc29a89-0d09-41dc-8c38-f8c272fd7689-openshift-service-ca\") pod \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " Sep 10 17:25:24.502616 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.502618 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-metrics-certs\") pod \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " Sep 10 17:25:24.502832 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.502787 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfqcv\" (UniqueName: \"kubernetes.io/projected/bcc29a89-0d09-41dc-8c38-f8c272fd7689-kube-api-access-dfqcv\") pod \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " Sep 10 17:25:24.502832 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.502816 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls\") pod \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\" (UID: \"bcc29a89-0d09-41dc-8c38-f8c272fd7689\") " Sep 10 17:25:24.502988 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.502966 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc29a89-0d09-41dc-8c38-f8c272fd7689-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcc29a89-0d09-41dc-8c38-f8c272fd7689" (UID: "bcc29a89-0d09-41dc-8c38-f8c272fd7689"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:25:24.503271 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.503238 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc29a89-0d09-41dc-8c38-f8c272fd7689-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:25:24.504929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.504902 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc29a89-0d09-41dc-8c38-f8c272fd7689-kube-api-access-dfqcv" (OuterVolumeSpecName: "kube-api-access-dfqcv") pod "bcc29a89-0d09-41dc-8c38-f8c272fd7689" (UID: "bcc29a89-0d09-41dc-8c38-f8c272fd7689"). InnerVolumeSpecName "kube-api-access-dfqcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:25:24.505034 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.504960 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bcc29a89-0d09-41dc-8c38-f8c272fd7689" (UID: "bcc29a89-0d09-41dc-8c38-f8c272fd7689"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:25:24.505034 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.504991 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcc29a89-0d09-41dc-8c38-f8c272fd7689" (UID: "bcc29a89-0d09-41dc-8c38-f8c272fd7689"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:25:24.580685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.580638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:24.603909 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.603842 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:25:24.603909 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.603875 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqcv\" (UniqueName: \"kubernetes.io/projected/bcc29a89-0d09-41dc-8c38-f8c272fd7689-kube-api-access-dfqcv\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:25:24.604152 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.603923 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc29a89-0d09-41dc-8c38-f8c272fd7689-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:25:24.718631 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.718531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:24.735400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.735364 2576 generic.go:356] "Generic (PLEG): container finished" podID="bcc29a89-0d09-41dc-8c38-f8c272fd7689" containerID="a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16" exitCode=0 Sep 10 17:25:24.736192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.735428 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" Sep 10 17:25:24.736192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.735465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" event={"ID":"bcc29a89-0d09-41dc-8c38-f8c272fd7689","Type":"ContainerDied","Data":"a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16"} Sep 10 17:25:24.736192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.735558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc" event={"ID":"bcc29a89-0d09-41dc-8c38-f8c272fd7689","Type":"ContainerDied","Data":"c7209413639705457c4687bef3602d2e56a7216e43e3fe5d10ffbb74a1874bc0"} Sep 10 17:25:24.736192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.735605 2576 scope.go:117] "RemoveContainer" containerID="a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16" Sep 10 17:25:24.753367 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.753304 2576 scope.go:117] "RemoveContainer" containerID="a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16" Sep 10 17:25:24.753696 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:25:24.753667 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16\": container with ID starting with a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16 not found: ID does not exist" containerID="a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16" Sep 10 17:25:24.753785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.753698 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16"} err="failed to get container status \"a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16\": rpc error: code = NotFound desc = could not find container \"a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16\": container with ID starting with a0dfb6d398eb739819c7de42bf5d96f56bc8f75b2a3c077fc9579d046c609f16 not found: ID does not exist" Sep 10 17:25:24.777025 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.776996 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc"] Sep 10 17:25:24.782665 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:24.782630 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99ff0092-85756d8f5c-m9bkc"] Sep 10 17:25:25.216799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:25.216759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:25.371912 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:25.371843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:25.510795 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:25.510705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:25.869031 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:25.868993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:26.051662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:26.051623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:26.249684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:26.249589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:26.487263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:26.487230 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc29a89-0d09-41dc-8c38-f8c272fd7689" path="/var/lib/kubelet/pods/bcc29a89-0d09-41dc-8c38-f8c272fd7689/volumes" Sep 10 17:25:28.470467 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.470400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:28.614095 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.614057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:28.727337 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.727238 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl"] Sep 10 17:25:28.727709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.727694 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc29a89-0d09-41dc-8c38-f8c272fd7689" containerName="maas-api" Sep 10 17:25:28.727709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.727707 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc29a89-0d09-41dc-8c38-f8c272fd7689" containerName="maas-api" Sep 10 17:25:28.727851 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.727783 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc29a89-0d09-41dc-8c38-f8c272fd7689" containerName="maas-api" Sep 10 17:25:28.732534 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.732513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.745657 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.745633 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-53de5224-serving-cert\"" Sep 10 17:25:28.760627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.760598 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl"] Sep 10 17:25:28.851156 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.851123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:28.852194 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.852164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-openshift-service-ca\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.852329 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.852285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-maas-api-tls\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.852373 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.852343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp7s\" (UniqueName: \"kubernetes.io/projected/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-kube-api-access-xfp7s\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.852413 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.852392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-metrics-certs\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.953792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.953750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-maas-api-tls\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.954037 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.953801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp7s\" (UniqueName: \"kubernetes.io/projected/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-kube-api-access-xfp7s\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.954037 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.953846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-metrics-certs\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.954037 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.953926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-openshift-service-ca\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.954767 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.954732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-openshift-service-ca\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.956378 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.956354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-maas-api-tls\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.956498 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.956480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-metrics-certs\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.978455 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.978376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp7s\" (UniqueName: \"kubernetes.io/projected/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-kube-api-access-xfp7s\") pod \"maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:28.992405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:28.992368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:29.043818 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.043781 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:29.346279 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.346248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl"] Sep 10 17:25:29.347152 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:25:29.347128 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454e63f2_4d96_4419_9bdd_7d3b75a6ed33.slice/crio-e0d09afe7dde9b9d115fa6ceb96dba08b4cabfcab8e16a82fa934344ce9bf67c WatchSource:0}: Error finding container e0d09afe7dde9b9d115fa6ceb96dba08b4cabfcab8e16a82fa934344ce9bf67c: Status 404 returned error can't find the container with id e0d09afe7dde9b9d115fa6ceb96dba08b4cabfcab8e16a82fa934344ce9bf67c Sep 10 17:25:29.693361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.693333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:29.770609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.770572 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" event={"ID":"454e63f2-4d96-4419-9bdd-7d3b75a6ed33","Type":"ContainerStarted","Data":"8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293"} Sep 10 17:25:29.770787 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.770626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" event={"ID":"454e63f2-4d96-4419-9bdd-7d3b75a6ed33","Type":"ContainerStarted","Data":"e0d09afe7dde9b9d115fa6ceb96dba08b4cabfcab8e16a82fa934344ce9bf67c"} Sep 10 17:25:29.770787 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.770778 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:29.801140 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.801086 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" podStartSLOduration=1.502093449 podStartE2EDuration="1.801068527s" totalImagesPullingTime="298.975078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:29.799364172 +0000 UTC m=+835.780444169" watchObservedRunningTime="2026-09-10 17:25:29.801068527 +0000 UTC m=+835.782148581" Sep 10 17:25:29.829638 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:29.829604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:30.789849 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:30.789809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:32.287557 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:32.287523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:35.782438 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:35.782406 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:53.103422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:53.103378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:59.678621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.678583 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl"] Sep 10 17:25:59.679155 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.678799 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" podUID="454e63f2-4d96-4419-9bdd-7d3b75a6ed33" containerName="maas-api" containerID="cri-o://8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293" gracePeriod=30 Sep 10 17:25:59.720502 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.720464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:59.842468 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.842426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:59.935581 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.935519 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:59.939967 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.939935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:25:59.942180 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.942157 2576 generic.go:356] "Generic (PLEG): container finished" podID="454e63f2-4d96-4419-9bdd-7d3b75a6ed33" containerID="8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293" exitCode=0 Sep 10 17:25:59.942305 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.942216 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" Sep 10 17:25:59.942305 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.942243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" event={"ID":"454e63f2-4d96-4419-9bdd-7d3b75a6ed33","Type":"ContainerDied","Data":"8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293"} Sep 10 17:25:59.942305 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.942295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl" event={"ID":"454e63f2-4d96-4419-9bdd-7d3b75a6ed33","Type":"ContainerDied","Data":"e0d09afe7dde9b9d115fa6ceb96dba08b4cabfcab8e16a82fa934344ce9bf67c"} Sep 10 17:25:59.942415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.942318 2576 scope.go:117] "RemoveContainer" containerID="8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293" Sep 10 17:25:59.953395 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.953376 2576 scope.go:117] "RemoveContainer" containerID="8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293" Sep 10 17:25:59.953700 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:25:59.953676 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293\": container with ID starting with 8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293 not found: ID does not exist" containerID="8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293" Sep 10 17:25:59.953789 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:25:59.953708 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293"} err="failed to get container status \"8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293\": rpc error: code = NotFound desc = could not find container \"8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293\": container with ID starting with 8010d349342368e5b1f587caecd8ab07daf8550f8478e56a87b21eee53eac293 not found: ID does not exist" Sep 10 17:26:00.065835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.065796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:00.070073 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.070040 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-metrics-certs\") pod \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " Sep 10 17:26:00.070231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.070108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-maas-api-tls\") pod \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " Sep 10 17:26:00.070231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.070142 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfp7s\" (UniqueName: \"kubernetes.io/projected/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-kube-api-access-xfp7s\") pod \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " Sep 10 17:26:00.070231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.070169 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-openshift-service-ca\") pod \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\" (UID: \"454e63f2-4d96-4419-9bdd-7d3b75a6ed33\") " Sep 10 17:26:00.070961 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.070926 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "454e63f2-4d96-4419-9bdd-7d3b75a6ed33" (UID: "454e63f2-4d96-4419-9bdd-7d3b75a6ed33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:26:00.072385 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.072357 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "454e63f2-4d96-4419-9bdd-7d3b75a6ed33" (UID: "454e63f2-4d96-4419-9bdd-7d3b75a6ed33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:26:00.072496 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.072379 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "454e63f2-4d96-4419-9bdd-7d3b75a6ed33" (UID: "454e63f2-4d96-4419-9bdd-7d3b75a6ed33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:26:00.072496 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.072360 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-kube-api-access-xfp7s" (OuterVolumeSpecName: "kube-api-access-xfp7s") pod "454e63f2-4d96-4419-9bdd-7d3b75a6ed33" (UID: "454e63f2-4d96-4419-9bdd-7d3b75a6ed33"). InnerVolumeSpecName "kube-api-access-xfp7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:26:00.162335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.162292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:00.171894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.171861 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:00.172046 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.171911 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfp7s\" (UniqueName: \"kubernetes.io/projected/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-kube-api-access-xfp7s\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:00.172046 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.171921 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:00.172046 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.171929 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454e63f2-4d96-4419-9bdd-7d3b75a6ed33-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:00.349274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.349237 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl"] Sep 10 17:26:00.364117 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.364090 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53de5224-697bff6c9c-qf7tl"] Sep 10 17:26:00.490724 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.490684 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454e63f2-4d96-4419-9bdd-7d3b75a6ed33" path="/var/lib/kubelet/pods/454e63f2-4d96-4419-9bdd-7d3b75a6ed33/volumes" Sep 10 17:26:00.491903 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.491824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:00.816552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:00.816508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:01.188656 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:01.188523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:01.292080 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:01.292033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:03.303853 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:03.303816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:03.411586 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:03.411530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:04.460375 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.460344 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf"] Sep 10 17:26:04.460804 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.460789 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454e63f2-4d96-4419-9bdd-7d3b75a6ed33" containerName="maas-api" Sep 10 17:26:04.460804 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.460805 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="454e63f2-4d96-4419-9bdd-7d3b75a6ed33" containerName="maas-api" Sep 10 17:26:04.460997 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.460923 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="454e63f2-4d96-4419-9bdd-7d3b75a6ed33" containerName="maas-api" Sep 10 17:26:04.464208 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.464192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.467568 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.467548 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4240652e-serving-cert\"" Sep 10 17:26:04.476831 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.476807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf"] Sep 10 17:26:04.581289 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.581243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:04.633001 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.632969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75zm\" (UniqueName: \"kubernetes.io/projected/d4fdba2d-0043-4128-8b29-55d204887a81-kube-api-access-t75zm\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.633189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.633092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-maas-api-tls\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.633295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.633226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdba2d-0043-4128-8b29-55d204887a81-openshift-service-ca\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.633513 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.633490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-metrics-certs\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.670280 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.670247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:04.734381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.734279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-maas-api-tls\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.734381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.734329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdba2d-0043-4128-8b29-55d204887a81-openshift-service-ca\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.734673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.734404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-metrics-certs\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.734673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.734442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75zm\" (UniqueName: \"kubernetes.io/projected/d4fdba2d-0043-4128-8b29-55d204887a81-kube-api-access-t75zm\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.735104 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.735079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdba2d-0043-4128-8b29-55d204887a81-openshift-service-ca\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.736801 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.736779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-metrics-certs\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.736922 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.736812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-maas-api-tls\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.743681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.743661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75zm\" (UniqueName: \"kubernetes.io/projected/d4fdba2d-0043-4128-8b29-55d204887a81-kube-api-access-t75zm\") pod \"maas-api-e2e-ait-4240652e-676f77d88-dzmxf\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.774686 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.774655 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:04.779154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:04.779125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:05.056200 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:05.056170 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf"] Sep 10 17:26:05.056752 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:26:05.056729 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fdba2d_0043_4128_8b29_55d204887a81.slice/crio-6c2b0f475312145b2d4846a73f6d8320c6712ed586e55e44514af4ac11695bee WatchSource:0}: Error finding container 6c2b0f475312145b2d4846a73f6d8320c6712ed586e55e44514af4ac11695bee: Status 404 returned error can't find the container with id 6c2b0f475312145b2d4846a73f6d8320c6712ed586e55e44514af4ac11695bee Sep 10 17:26:05.179678 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:05.179637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:05.282393 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:05.282359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:05.980006 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:05.979909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" event={"ID":"d4fdba2d-0043-4128-8b29-55d204887a81","Type":"ContainerStarted","Data":"0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f"} Sep 10 17:26:05.980006 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:05.979960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" event={"ID":"d4fdba2d-0043-4128-8b29-55d204887a81","Type":"ContainerStarted","Data":"6c2b0f475312145b2d4846a73f6d8320c6712ed586e55e44514af4ac11695bee"} Sep 10 17:26:05.980426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:05.980138 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:06.006555 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:06.006503 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" podStartSLOduration=1.487743695 podStartE2EDuration="2.006484808s" totalImagesPullingTime="518.741113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:06.003163273 +0000 UTC m=+871.984243273" watchObservedRunningTime="2026-09-10 17:26:06.006484808 +0000 UTC m=+871.987564805" Sep 10 17:26:09.927556 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:09.927513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:11.990703 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:11.990668 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:18.938449 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:18.938392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:34.488680 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:34.488608 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:26:34.493167 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:34.493142 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:26:35.173975 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.173945 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf"] Sep 10 17:26:35.174189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.174140 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" podUID="d4fdba2d-0043-4128-8b29-55d204887a81" containerName="maas-api" containerID="cri-o://0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f" gracePeriod=30 Sep 10 17:26:35.225532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.225483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:35.338495 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.338457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:35.421603 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.421574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:35.426178 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.426126 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:35.532838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.532802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:35.548571 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.548548 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t75zm\" (UniqueName: \"kubernetes.io/projected/d4fdba2d-0043-4128-8b29-55d204887a81-kube-api-access-t75zm\") pod \"d4fdba2d-0043-4128-8b29-55d204887a81\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " Sep 10 17:26:35.548684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.548592 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-maas-api-tls\") pod \"d4fdba2d-0043-4128-8b29-55d204887a81\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " Sep 10 17:26:35.548684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.548638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdba2d-0043-4128-8b29-55d204887a81-openshift-service-ca\") pod \"d4fdba2d-0043-4128-8b29-55d204887a81\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " Sep 10 17:26:35.548684 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.548658 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-metrics-certs\") pod \"d4fdba2d-0043-4128-8b29-55d204887a81\" (UID: \"d4fdba2d-0043-4128-8b29-55d204887a81\") " Sep 10 17:26:35.549068 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.549042 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fdba2d-0043-4128-8b29-55d204887a81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4fdba2d-0043-4128-8b29-55d204887a81" (UID: "d4fdba2d-0043-4128-8b29-55d204887a81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:26:35.550803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.550769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4fdba2d-0043-4128-8b29-55d204887a81" (UID: "d4fdba2d-0043-4128-8b29-55d204887a81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:26:35.550803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.550791 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d4fdba2d-0043-4128-8b29-55d204887a81" (UID: "d4fdba2d-0043-4128-8b29-55d204887a81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:26:35.550960 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.550798 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fdba2d-0043-4128-8b29-55d204887a81-kube-api-access-t75zm" (OuterVolumeSpecName: "kube-api-access-t75zm") pod "d4fdba2d-0043-4128-8b29-55d204887a81" (UID: "d4fdba2d-0043-4128-8b29-55d204887a81"). InnerVolumeSpecName "kube-api-access-t75zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:26:35.634363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.634321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:35.650089 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.650055 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t75zm\" (UniqueName: \"kubernetes.io/projected/d4fdba2d-0043-4128-8b29-55d204887a81-kube-api-access-t75zm\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:35.650089 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.650086 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:35.650283 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.650098 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fdba2d-0043-4128-8b29-55d204887a81-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:35.650283 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:35.650106 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fdba2d-0043-4128-8b29-55d204887a81-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:36.164643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.164602 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4fdba2d-0043-4128-8b29-55d204887a81" containerID="0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f" exitCode=0 Sep 10 17:26:36.164855 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.164658 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" Sep 10 17:26:36.164855 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.164686 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" event={"ID":"d4fdba2d-0043-4128-8b29-55d204887a81","Type":"ContainerDied","Data":"0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f"} Sep 10 17:26:36.164855 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.164735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf" event={"ID":"d4fdba2d-0043-4128-8b29-55d204887a81","Type":"ContainerDied","Data":"6c2b0f475312145b2d4846a73f6d8320c6712ed586e55e44514af4ac11695bee"} Sep 10 17:26:36.164855 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.164757 2576 scope.go:117] "RemoveContainer" containerID="0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f" Sep 10 17:26:36.174931 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.174901 2576 scope.go:117] "RemoveContainer" containerID="0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f" Sep 10 17:26:36.175211 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:26:36.175191 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f\": container with ID starting with 0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f not found: ID does not exist" containerID="0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f" Sep 10 17:26:36.175279 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.175219 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f"} err="failed to get container status \"0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f\": rpc error: code = NotFound desc = could not find container \"0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f\": container with ID starting with 0f6c48a868c3a2b2441a245d1f24e4f68e0f4d1005af1309ec60ccac2959ef0f not found: ID does not exist" Sep 10 17:26:36.190994 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.190952 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf"] Sep 10 17:26:36.199615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.199581 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4240652e-676f77d88-dzmxf"] Sep 10 17:26:36.487160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.487079 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fdba2d-0043-4128-8b29-55d204887a81" path="/var/lib/kubelet/pods/d4fdba2d-0043-4128-8b29-55d204887a81/volumes" Sep 10 17:26:36.766449 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:36.766359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:37.164305 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:37.164266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:37.541319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:37.541222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:37.727399 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:37.727357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:40.552401 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:40.552367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:40.650499 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:40.650465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:41.789890 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.789835 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp"] Sep 10 17:26:41.790428 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.790408 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fdba2d-0043-4128-8b29-55d204887a81" containerName="maas-api" Sep 10 17:26:41.790490 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.790433 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fdba2d-0043-4128-8b29-55d204887a81" containerName="maas-api" Sep 10 17:26:41.790595 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.790583 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fdba2d-0043-4128-8b29-55d204887a81" containerName="maas-api" Sep 10 17:26:41.794953 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.794932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:41.797990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.797961 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f4c26bd5-serving-cert\"" Sep 10 17:26:41.810191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.810168 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp"] Sep 10 17:26:41.843332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.843300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:41.915383 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.915352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57w5\" (UniqueName: \"kubernetes.io/projected/173940a1-e3fb-4cd0-85de-cf201b4213af-kube-api-access-v57w5\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:41.915383 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.915387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:41.915617 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.915406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173940a1-e3fb-4cd0-85de-cf201b4213af-openshift-service-ca\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:41.915617 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.915543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-metrics-certs\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:41.937996 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:41.937962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:42.016273 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.016231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57w5\" (UniqueName: \"kubernetes.io/projected/173940a1-e3fb-4cd0-85de-cf201b4213af-kube-api-access-v57w5\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.016273 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.016277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.016500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.016296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173940a1-e3fb-4cd0-85de-cf201b4213af-openshift-service-ca\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.016500 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:26:42.016401 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-serving-cert: secret "maas-api-e2e-derive-f4c26bd5-serving-cert" not found Sep 10 17:26:42.016500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.016434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-metrics-certs\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.016500 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:26:42.016481 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls podName:173940a1-e3fb-4cd0-85de-cf201b4213af nodeName:}" failed. No retries permitted until 2026-09-10 17:26:42.516457567 +0000 UTC m=+908.497537566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls") pod "maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" (UID: "173940a1-e3fb-4cd0-85de-cf201b4213af") : secret "maas-api-e2e-derive-f4c26bd5-serving-cert" not found Sep 10 17:26:42.017032 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.017006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173940a1-e3fb-4cd0-85de-cf201b4213af-openshift-service-ca\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.018822 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.018791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-metrics-certs\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.030481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.030454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57w5\" (UniqueName: \"kubernetes.io/projected/173940a1-e3fb-4cd0-85de-cf201b4213af-kube-api-access-v57w5\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.133067 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.133032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:42.521291 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.521203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.523650 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.523613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls\") pod \"maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.544411 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.544383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:42.641004 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.640968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:42.705778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.705741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:42.987730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:42.987706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp"] Sep 10 17:26:42.989639 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:26:42.989616 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173940a1_e3fb_4cd0_85de_cf201b4213af.slice/crio-bf142f76e01fe37673204085845b4b03e3ea31d4d136ef55f7120c6bde979d22 WatchSource:0}: Error finding container bf142f76e01fe37673204085845b4b03e3ea31d4d136ef55f7120c6bde979d22: Status 404 returned error can't find the container with id bf142f76e01fe37673204085845b4b03e3ea31d4d136ef55f7120c6bde979d22 Sep 10 17:26:43.206340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:43.206306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" event={"ID":"173940a1-e3fb-4cd0-85de-cf201b4213af","Type":"ContainerStarted","Data":"bf142f76e01fe37673204085845b4b03e3ea31d4d136ef55f7120c6bde979d22"} Sep 10 17:26:44.213456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:44.213425 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" event={"ID":"173940a1-e3fb-4cd0-85de-cf201b4213af","Type":"ContainerStarted","Data":"904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e"} Sep 10 17:26:44.213852 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:44.213480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:44.236235 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:44.236179 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" podStartSLOduration=2.845758516 podStartE2EDuration="3.236163402s" totalImagesPullingTime="390.404886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:44.234176332 +0000 UTC m=+910.215256331" watchObservedRunningTime="2026-09-10 17:26:44.236163402 +0000 UTC m=+910.217243400" Sep 10 17:26:45.917509 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:45.917468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:50.225307 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:50.225275 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:26:56.493860 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.493819 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv"] Sep 10 17:26:56.494314 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.494028 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" podUID="4afd73a8-5881-4e80-9261-356dbd2982d1" containerName="maas-api" containerID="cri-o://1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf" gracePeriod=30 Sep 10 17:26:56.516946 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.516917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:56.634864 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.634816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:56.735644 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.735605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:56.753079 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.752964 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:26:56.843461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.837159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:56.856068 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.856024 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls\") pod \"4afd73a8-5881-4e80-9261-356dbd2982d1\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " Sep 10 17:26:56.856225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.856100 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-metrics-certs\") pod \"4afd73a8-5881-4e80-9261-356dbd2982d1\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " Sep 10 17:26:56.856225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.856129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-742vr\" (UniqueName: \"kubernetes.io/projected/4afd73a8-5881-4e80-9261-356dbd2982d1-kube-api-access-742vr\") pod \"4afd73a8-5881-4e80-9261-356dbd2982d1\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " Sep 10 17:26:56.856302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.856251 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afd73a8-5881-4e80-9261-356dbd2982d1-openshift-service-ca\") pod \"4afd73a8-5881-4e80-9261-356dbd2982d1\" (UID: \"4afd73a8-5881-4e80-9261-356dbd2982d1\") " Sep 10 17:26:56.856619 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.856589 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afd73a8-5881-4e80-9261-356dbd2982d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4afd73a8-5881-4e80-9261-356dbd2982d1" (UID: "4afd73a8-5881-4e80-9261-356dbd2982d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:26:56.858224 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.858199 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afd73a8-5881-4e80-9261-356dbd2982d1-kube-api-access-742vr" (OuterVolumeSpecName: "kube-api-access-742vr") pod "4afd73a8-5881-4e80-9261-356dbd2982d1" (UID: "4afd73a8-5881-4e80-9261-356dbd2982d1"). InnerVolumeSpecName "kube-api-access-742vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:26:56.858614 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.858588 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4afd73a8-5881-4e80-9261-356dbd2982d1" (UID: "4afd73a8-5881-4e80-9261-356dbd2982d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:26:56.858614 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.858601 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4afd73a8-5881-4e80-9261-356dbd2982d1" (UID: "4afd73a8-5881-4e80-9261-356dbd2982d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:26:56.933599 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.933562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:56.957380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.957343 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4afd73a8-5881-4e80-9261-356dbd2982d1-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:56.957380 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.957374 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:56.957565 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.957387 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4afd73a8-5881-4e80-9261-356dbd2982d1-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:56.957565 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:56.957400 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742vr\" (UniqueName: \"kubernetes.io/projected/4afd73a8-5881-4e80-9261-356dbd2982d1-kube-api-access-742vr\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:26:57.239070 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.239022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:26:57.289744 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.289709 2576 generic.go:356] "Generic (PLEG): container finished" podID="4afd73a8-5881-4e80-9261-356dbd2982d1" containerID="1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf" exitCode=0 Sep 10 17:26:57.289982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.289765 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" Sep 10 17:26:57.289982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.289793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" event={"ID":"4afd73a8-5881-4e80-9261-356dbd2982d1","Type":"ContainerDied","Data":"1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf"} Sep 10 17:26:57.289982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.289872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv" event={"ID":"4afd73a8-5881-4e80-9261-356dbd2982d1","Type":"ContainerDied","Data":"52f2fcd48fffe51c4f8a34ac1e9c907414101b1eb2748d0eb1110d9b88fe2059"} Sep 10 17:26:57.289982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.289911 2576 scope.go:117] "RemoveContainer" containerID="1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf" Sep 10 17:26:57.299918 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.299903 2576 scope.go:117] "RemoveContainer" containerID="1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf" Sep 10 17:26:57.300217 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:26:57.300197 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf\": container with ID starting with 1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf not found: ID does not exist" containerID="1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf" Sep 10 17:26:57.300267 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.300226 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf"} err="failed to get container status \"1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf\": rpc error: code = NotFound desc = could not find container \"1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf\": container with ID starting with 1704ad3cbaf60c3f1b8d91cf3358c29b15b71020f7c9b19ed3a9cf310e6bbfcf not found: ID does not exist" Sep 10 17:26:57.316836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.316808 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv"] Sep 10 17:26:57.319374 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:57.319355 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-889898-dc67cb7f-q6xrv"] Sep 10 17:26:58.486896 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:26:58.486852 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afd73a8-5881-4e80-9261-356dbd2982d1" path="/var/lib/kubelet/pods/4afd73a8-5881-4e80-9261-356dbd2982d1/volumes" Sep 10 17:27:02.305757 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:02.305721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:03.045659 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:03.045623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:06.110476 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:06.110438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:06.902071 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:06.902029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:07.898563 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:07.898522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:09.833775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:09.833741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:12.666866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:12.666442 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp"] Sep 10 17:27:12.666866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:12.666687 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" podUID="173940a1-e3fb-4cd0-85de-cf201b4213af" containerName="maas-api" containerID="cri-o://904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e" gracePeriod=30 Sep 10 17:27:12.711148 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:12.711064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:12.794706 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:12.794670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:12.872811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:12.872766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:12.928524 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:12.928496 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:27:12.964212 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:12.964122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:13.018104 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.018070 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls\") pod \"173940a1-e3fb-4cd0-85de-cf201b4213af\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " Sep 10 17:27:13.018278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.018121 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-metrics-certs\") pod \"173940a1-e3fb-4cd0-85de-cf201b4213af\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " Sep 10 17:27:13.018278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.018197 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173940a1-e3fb-4cd0-85de-cf201b4213af-openshift-service-ca\") pod \"173940a1-e3fb-4cd0-85de-cf201b4213af\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " Sep 10 17:27:13.018278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.018222 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v57w5\" (UniqueName: \"kubernetes.io/projected/173940a1-e3fb-4cd0-85de-cf201b4213af-kube-api-access-v57w5\") pod \"173940a1-e3fb-4cd0-85de-cf201b4213af\" (UID: \"173940a1-e3fb-4cd0-85de-cf201b4213af\") " Sep 10 17:27:13.018563 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.018540 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173940a1-e3fb-4cd0-85de-cf201b4213af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "173940a1-e3fb-4cd0-85de-cf201b4213af" (UID: "173940a1-e3fb-4cd0-85de-cf201b4213af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:27:13.020394 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.020370 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "173940a1-e3fb-4cd0-85de-cf201b4213af" (UID: "173940a1-e3fb-4cd0-85de-cf201b4213af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:27:13.020492 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.020387 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173940a1-e3fb-4cd0-85de-cf201b4213af-kube-api-access-v57w5" (OuterVolumeSpecName: "kube-api-access-v57w5") pod "173940a1-e3fb-4cd0-85de-cf201b4213af" (UID: "173940a1-e3fb-4cd0-85de-cf201b4213af"). InnerVolumeSpecName "kube-api-access-v57w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:27:13.020492 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.020401 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "173940a1-e3fb-4cd0-85de-cf201b4213af" (UID: "173940a1-e3fb-4cd0-85de-cf201b4213af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:27:13.037457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.037424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:13.119786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.119753 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/173940a1-e3fb-4cd0-85de-cf201b4213af-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:27:13.119786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.119782 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57w5\" (UniqueName: \"kubernetes.io/projected/173940a1-e3fb-4cd0-85de-cf201b4213af-kube-api-access-v57w5\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:27:13.119786 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.119792 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:27:13.120102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.119800 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/173940a1-e3fb-4cd0-85de-cf201b4213af-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:27:13.141379 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.141348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:13.390997 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.390966 2576 generic.go:356] "Generic (PLEG): container finished" podID="173940a1-e3fb-4cd0-85de-cf201b4213af" containerID="904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e" exitCode=0 Sep 10 17:27:13.391211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.391010 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" Sep 10 17:27:13.391211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.391054 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" event={"ID":"173940a1-e3fb-4cd0-85de-cf201b4213af","Type":"ContainerDied","Data":"904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e"} Sep 10 17:27:13.391211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.391113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp" event={"ID":"173940a1-e3fb-4cd0-85de-cf201b4213af","Type":"ContainerDied","Data":"bf142f76e01fe37673204085845b4b03e3ea31d4d136ef55f7120c6bde979d22"} Sep 10 17:27:13.391211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.391146 2576 scope.go:117] "RemoveContainer" containerID="904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e" Sep 10 17:27:13.401337 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.401321 2576 scope.go:117] "RemoveContainer" containerID="904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e" Sep 10 17:27:13.401599 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:27:13.401581 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e\": container with ID starting with 904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e not found: ID does not exist" containerID="904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e" Sep 10 17:27:13.401639 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.401610 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e"} err="failed to get container status \"904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e\": rpc error: code = NotFound desc = could not find container \"904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e\": container with ID starting with 904299aef546e3bbae4e0e7a143899a884b7af15f0f14a73895fefa68aafae9e not found: ID does not exist" Sep 10 17:27:13.417377 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.417348 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp"] Sep 10 17:27:13.423347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:13.423326 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f4c26bd5-64547f8c45-7b8kp"] Sep 10 17:27:14.488792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:14.488757 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173940a1-e3fb-4cd0-85de-cf201b4213af" path="/var/lib/kubelet/pods/173940a1-e3fb-4cd0-85de-cf201b4213af/volumes" Sep 10 17:27:14.558345 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:14.558302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:15.512035 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:15.511992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:16.667790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:16.667753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:18.541568 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:18.541527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:21.579325 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:21.579288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:21.689308 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:21.689277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:21.801413 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:21.801380 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:21.908293 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:21.908199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:22.298230 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:22.298134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:36.141049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:36.141005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:41.345290 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:41.345247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:41.558822 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:41.558782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:41.962658 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:41.962623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:42.409481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:42.409443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:42.769269 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:42.769173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:45.530949 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:45.530868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:45.694372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:45.694330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:46.820156 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.820125 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l"] Sep 10 17:27:46.820660 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.820640 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173940a1-e3fb-4cd0-85de-cf201b4213af" containerName="maas-api" Sep 10 17:27:46.820715 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.820664 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="173940a1-e3fb-4cd0-85de-cf201b4213af" containerName="maas-api" Sep 10 17:27:46.820715 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.820690 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afd73a8-5881-4e80-9261-356dbd2982d1" containerName="maas-api" Sep 10 17:27:46.820715 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.820699 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afd73a8-5881-4e80-9261-356dbd2982d1" containerName="maas-api" Sep 10 17:27:46.820838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.820786 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afd73a8-5881-4e80-9261-356dbd2982d1" containerName="maas-api" Sep 10 17:27:46.820838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.820798 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="173940a1-e3fb-4cd0-85de-cf201b4213af" containerName="maas-api" Sep 10 17:27:46.825290 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.825270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:46.830944 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.830922 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4b53b5-serving-cert\"" Sep 10 17:27:46.844189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.844156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l"] Sep 10 17:27:46.932607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.932567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:46.955902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.955855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02e2863-43e4-4322-b624-f9bc56dfa3b3-openshift-service-ca\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:46.956065 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.955915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-metrics-certs\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:46.956065 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.955937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-maas-api-tls\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:46.956065 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:46.955979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f48hl\" (UniqueName: \"kubernetes.io/projected/d02e2863-43e4-4322-b624-f9bc56dfa3b3-kube-api-access-f48hl\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.050442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.050401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:47.057259 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.057226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02e2863-43e4-4322-b624-f9bc56dfa3b3-openshift-service-ca\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.057400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.057275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-metrics-certs\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.057400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.057302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-maas-api-tls\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.057400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.057355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f48hl\" (UniqueName: \"kubernetes.io/projected/d02e2863-43e4-4322-b624-f9bc56dfa3b3-kube-api-access-f48hl\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.058004 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.057984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02e2863-43e4-4322-b624-f9bc56dfa3b3-openshift-service-ca\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.059809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.059789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-metrics-certs\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.059949 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.059836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-maas-api-tls\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.070850 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.070786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f48hl\" (UniqueName: \"kubernetes.io/projected/d02e2863-43e4-4322-b624-f9bc56dfa3b3-kube-api-access-f48hl\") pod \"maas-api-e2e-explicit-4b53b5-867df6844c-nq45l\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.136226 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.136185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:47.430791 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.430764 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l"] Sep 10 17:27:47.432470 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:27:47.432437 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02e2863_43e4_4322_b624_f9bc56dfa3b3.slice/crio-aa60c003d180ec2ea651d0d37a362cd8aa1f193e17b55f222a7a625fa71d8ff1 WatchSource:0}: Error finding container aa60c003d180ec2ea651d0d37a362cd8aa1f193e17b55f222a7a625fa71d8ff1: Status 404 returned error can't find the container with id aa60c003d180ec2ea651d0d37a362cd8aa1f193e17b55f222a7a625fa71d8ff1 Sep 10 17:27:47.433795 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.433775 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:27:47.597951 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:47.597912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" event={"ID":"d02e2863-43e4-4322-b624-f9bc56dfa3b3","Type":"ContainerStarted","Data":"aa60c003d180ec2ea651d0d37a362cd8aa1f193e17b55f222a7a625fa71d8ff1"} Sep 10 17:27:48.043541 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:48.043491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:48.448219 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:48.448178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:48.605074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:48.605037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" event={"ID":"d02e2863-43e4-4322-b624-f9bc56dfa3b3","Type":"ContainerStarted","Data":"486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce"} Sep 10 17:27:48.605247 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:48.605105 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:48.649809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:48.649769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:54.617275 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:54.617242 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:27:54.645897 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:54.645838 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" podStartSLOduration=8.263625005 podStartE2EDuration="8.645815474s" totalImagesPullingTime="382.190469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:48.643106225 +0000 UTC m=+974.624186222" watchObservedRunningTime="2026-09-10 17:27:54.645815474 +0000 UTC m=+980.626895522" Sep 10 17:27:56.045265 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:56.045232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:56.251010 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:56.250974 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:27:56.367927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:27:56.367866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:12.219372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:12.219280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:15.119769 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:15.119733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:15.523117 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:15.523035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:18.417249 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:18.417206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:18.602245 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:18.602211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:18.728301 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:18.728207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:24.594332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:24.594294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:29.146701 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:29.146658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:29.930322 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:29.930283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:37.277149 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:37.277113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:37.475262 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:37.475228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:37.612937 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:37.612900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:38.354078 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:38.354046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:38.496198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:38.496162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:40.222554 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:40.222522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:41.103537 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:41.103500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:47.805760 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:47.805722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:47.904559 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:47.904526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:48.617252 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:48.617214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:50.060935 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:50.060896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:51.002471 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:51.002436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:51.240459 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:51.240418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:52.062799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.062754 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l"] Sep 10 17:28:52.063043 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.063017 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" podUID="d02e2863-43e4-4322-b624-f9bc56dfa3b3" containerName="maas-api" containerID="cri-o://486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce" gracePeriod=30 Sep 10 17:28:52.114955 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.114873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:52.233597 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.233559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:52.314244 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.314174 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:28:52.373318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.373276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:52.403908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.403850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-maas-api-tls\") pod \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " Sep 10 17:28:52.404074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.403971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f48hl\" (UniqueName: \"kubernetes.io/projected/d02e2863-43e4-4322-b624-f9bc56dfa3b3-kube-api-access-f48hl\") pod \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " Sep 10 17:28:52.404074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.404004 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02e2863-43e4-4322-b624-f9bc56dfa3b3-openshift-service-ca\") pod \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " Sep 10 17:28:52.404074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.404049 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-metrics-certs\") pod \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\" (UID: \"d02e2863-43e4-4322-b624-f9bc56dfa3b3\") " Sep 10 17:28:52.404414 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.404390 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02e2863-43e4-4322-b624-f9bc56dfa3b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d02e2863-43e4-4322-b624-f9bc56dfa3b3" (UID: "d02e2863-43e4-4322-b624-f9bc56dfa3b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:28:52.404519 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.404472 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02e2863-43e4-4322-b624-f9bc56dfa3b3-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:28:52.406453 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.406418 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d02e2863-43e4-4322-b624-f9bc56dfa3b3" (UID: "d02e2863-43e4-4322-b624-f9bc56dfa3b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:28:52.406572 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.406434 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d02e2863-43e4-4322-b624-f9bc56dfa3b3" (UID: "d02e2863-43e4-4322-b624-f9bc56dfa3b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:28:52.406572 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.406417 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02e2863-43e4-4322-b624-f9bc56dfa3b3-kube-api-access-f48hl" (OuterVolumeSpecName: "kube-api-access-f48hl") pod "d02e2863-43e4-4322-b624-f9bc56dfa3b3" (UID: "d02e2863-43e4-4322-b624-f9bc56dfa3b3"). InnerVolumeSpecName "kube-api-access-f48hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:28:52.505201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.505167 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f48hl\" (UniqueName: \"kubernetes.io/projected/d02e2863-43e4-4322-b624-f9bc56dfa3b3-kube-api-access-f48hl\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:28:52.505201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.505204 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:28:52.505381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.505218 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02e2863-43e4-4322-b624-f9bc56dfa3b3-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:28:52.521373 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.521334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:52.775643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.775605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:52.907164 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.907126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:52.978287 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.978250 2576 generic.go:356] "Generic (PLEG): container finished" podID="d02e2863-43e4-4322-b624-f9bc56dfa3b3" containerID="486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce" exitCode=0 Sep 10 17:28:52.978456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.978295 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" Sep 10 17:28:52.978456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.978328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" event={"ID":"d02e2863-43e4-4322-b624-f9bc56dfa3b3","Type":"ContainerDied","Data":"486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce"} Sep 10 17:28:52.978456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.978381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l" event={"ID":"d02e2863-43e4-4322-b624-f9bc56dfa3b3","Type":"ContainerDied","Data":"aa60c003d180ec2ea651d0d37a362cd8aa1f193e17b55f222a7a625fa71d8ff1"} Sep 10 17:28:52.978456 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.978406 2576 scope.go:117] "RemoveContainer" containerID="486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce" Sep 10 17:28:52.987775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.987757 2576 scope.go:117] "RemoveContainer" containerID="486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce" Sep 10 17:28:52.988086 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:28:52.988062 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce\": container with ID starting with 486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce not found: ID does not exist" containerID="486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce" Sep 10 17:28:52.988211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:52.988098 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce"} err="failed to get container status \"486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce\": rpc error: code = NotFound desc = could not find container \"486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce\": container with ID starting with 486a058e2a59d37f8e73af108c43ff1359d765725ecb2336f06a0d51e8d3adce not found: ID does not exist" Sep 10 17:28:53.001152 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:53.001121 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l"] Sep 10 17:28:53.009892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:53.009857 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b53b5-867df6844c-nq45l"] Sep 10 17:28:54.487106 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:54.487070 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02e2863-43e4-4322-b624-f9bc56dfa3b3" path="/var/lib/kubelet/pods/d02e2863-43e4-4322-b624-f9bc56dfa3b3/volumes" Sep 10 17:28:56.484631 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:56.484589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:56.623405 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:56.623372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:56.731931 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:56.731901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:57.093425 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:57.093387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:28:57.318315 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:28:57.318277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:00.002072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:00.002037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:00.123979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:00.123941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:00.265329 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:00.265233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:00.372817 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:00.372786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:00.546051 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:00.545963 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:01.007187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.007152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:01.182033 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.181993 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr"] Sep 10 17:29:01.182431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.182414 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02e2863-43e4-4322-b624-f9bc56dfa3b3" containerName="maas-api" Sep 10 17:29:01.182431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.182430 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02e2863-43e4-4322-b624-f9bc56dfa3b3" containerName="maas-api" Sep 10 17:29:01.182642 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.182506 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02e2863-43e4-4322-b624-f9bc56dfa3b3" containerName="maas-api" Sep 10 17:29:01.187249 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.187220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.190255 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.190230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6dc6af55-serving-cert\"" Sep 10 17:29:01.218375 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.218338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr"] Sep 10 17:29:01.262845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.262759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:01.288664 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.288622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-metrics-certs\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.288838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.288677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.288838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.288785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54857\" (UniqueName: \"kubernetes.io/projected/2ae8107e-80d4-491a-b889-6c2fce25f10e-kube-api-access-54857\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.289010 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.288841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8107e-80d4-491a-b889-6c2fce25f10e-openshift-service-ca\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.389453 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.389409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-metrics-certs\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.389655 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.389550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.389655 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.389625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54857\" (UniqueName: \"kubernetes.io/projected/2ae8107e-80d4-491a-b889-6c2fce25f10e-kube-api-access-54857\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.389780 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.389683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8107e-80d4-491a-b889-6c2fce25f10e-openshift-service-ca\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.389780 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:01.389716 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-serving-cert: secret "maas-api-e2e-mt-6dc6af55-serving-cert" not found Sep 10 17:29:01.389918 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:01.389795 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls podName:2ae8107e-80d4-491a-b889-6c2fce25f10e nodeName:}" failed. No retries permitted until 2026-09-10 17:29:01.889772252 +0000 UTC m=+1047.870852238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls") pod "maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" (UID: "2ae8107e-80d4-491a-b889-6c2fce25f10e") : secret "maas-api-e2e-mt-6dc6af55-serving-cert" not found Sep 10 17:29:01.390531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.390505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8107e-80d4-491a-b889-6c2fce25f10e-openshift-service-ca\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.392052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.392023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-metrics-certs\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.401188 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.401156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:01.408367 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.408335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54857\" (UniqueName: \"kubernetes.io/projected/2ae8107e-80d4-491a-b889-6c2fce25f10e-kube-api-access-54857\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.520107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.520019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:01.867355 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.867323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:01.895796 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.895759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:01.898254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:01.898232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls\") pod \"maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:02.113838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.113747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:02.140174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.140140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:02.298608 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.298576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:02.439751 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.439395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr"] Sep 10 17:29:02.439964 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:29:02.439923 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae8107e_80d4_491a_b889_6c2fce25f10e.slice/crio-9f3e396b30dd866c479b865354339b14ef3b1b9ea27e5e7021eaff2d23c03f13 WatchSource:0}: Error finding container 9f3e396b30dd866c479b865354339b14ef3b1b9ea27e5e7021eaff2d23c03f13: Status 404 returned error can't find the container with id 9f3e396b30dd866c479b865354339b14ef3b1b9ea27e5e7021eaff2d23c03f13 Sep 10 17:29:02.591354 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.591318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:02.806717 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.806634 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd"] Sep 10 17:29:02.806875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.806829 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" podUID="8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" containerName="maas-api" containerID="cri-o://135fcd64b4099fcb411623fe63db3923e30c23602c5d7daaaaed7362c3103916" gracePeriod=30 Sep 10 17:29:02.895403 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:02.895367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:03.049645 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.049590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" event={"ID":"2ae8107e-80d4-491a-b889-6c2fce25f10e","Type":"ContainerStarted","Data":"9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b"} Sep 10 17:29:03.049816 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.049654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" event={"ID":"2ae8107e-80d4-491a-b889-6c2fce25f10e","Type":"ContainerStarted","Data":"9f3e396b30dd866c479b865354339b14ef3b1b9ea27e5e7021eaff2d23c03f13"} Sep 10 17:29:03.049816 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.049688 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:03.051134 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.051110 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" containerID="135fcd64b4099fcb411623fe63db3923e30c23602c5d7daaaaed7362c3103916" exitCode=0 Sep 10 17:29:03.051220 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.051175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" event={"ID":"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e","Type":"ContainerDied","Data":"135fcd64b4099fcb411623fe63db3923e30c23602c5d7daaaaed7362c3103916"} Sep 10 17:29:03.069846 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.069826 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:29:03.071537 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.071511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:03.093909 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.093847 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" podStartSLOduration=1.682739653 podStartE2EDuration="2.093828269s" totalImagesPullingTime="411.088616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:03.091236666 +0000 UTC m=+1049.072316664" watchObservedRunningTime="2026-09-10 17:29:03.093828269 +0000 UTC m=+1049.074908266" Sep 10 17:29:03.211397 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.211355 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-metrics-certs\") pod \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " Sep 10 17:29:03.211845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.211511 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls\") pod \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " Sep 10 17:29:03.211845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.211549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kz2d\" (UniqueName: \"kubernetes.io/projected/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-kube-api-access-4kz2d\") pod \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " Sep 10 17:29:03.211845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.211651 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-openshift-service-ca\") pod \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\" (UID: \"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e\") " Sep 10 17:29:03.212108 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.212074 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" (UID: "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:29:03.213641 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.213614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" (UID: "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:29:03.213641 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.213631 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" (UID: "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:29:03.213770 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.213666 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-kube-api-access-4kz2d" (OuterVolumeSpecName: "kube-api-access-4kz2d") pod "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" (UID: "8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e"). InnerVolumeSpecName "kube-api-access-4kz2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:29:03.285577 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.285502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:03.314902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.312924 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:03.314902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.312956 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kz2d\" (UniqueName: \"kubernetes.io/projected/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-kube-api-access-4kz2d\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:03.314902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.312968 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:03.314902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.312980 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:03.470647 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.470616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:03.670275 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.670235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:03.800636 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:03.800539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:04.051848 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:04.051754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:04.057910 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:04.057862 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" Sep 10 17:29:04.058065 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:04.057911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd" event={"ID":"8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e","Type":"ContainerDied","Data":"b81dc9c297acfd1ec997dd3402d711c08499fef727a253c27a56ae7ab205f71b"} Sep 10 17:29:04.058065 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:04.057971 2576 scope.go:117] "RemoveContainer" containerID="135fcd64b4099fcb411623fe63db3923e30c23602c5d7daaaaed7362c3103916" Sep 10 17:29:04.106172 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:04.106137 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd"] Sep 10 17:29:04.119234 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:04.119209 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc5a37-7c987d4ddb-wjtqd"] Sep 10 17:29:04.487748 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:04.487712 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" path="/var/lib/kubelet/pods/8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e/volumes" Sep 10 17:29:09.063384 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:09.063356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:09.120567 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:09.120533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:11.555382 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:11.555345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:11.656037 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:11.656005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:11.831865 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:11.831831 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r"] Sep 10 17:29:11.832136 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:11.832110 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" containerID="cri-o://1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002" gracePeriod=30 Sep 10 17:29:11.886520 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:11.886484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:12.382700 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:12.382658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:17.099951 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.099894 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:29:17.141119 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.141083 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerID="1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002" exitCode=0 Sep 10 17:29:17.141278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.141140 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" Sep 10 17:29:17.141278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.141163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" event={"ID":"1a5633e8-f77a-4313-a1f8-9fd99bf8b478","Type":"ContainerDied","Data":"1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002"} Sep 10 17:29:17.141278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.141216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r" event={"ID":"1a5633e8-f77a-4313-a1f8-9fd99bf8b478","Type":"ContainerDied","Data":"e5e886abcb5ba7f29436b5a21a7db98b4dbd1572fa9d0008ef097bea0f01c12a"} Sep 10 17:29:17.141278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.141228 2576 scope.go:117] "RemoveContainer" containerID="1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002" Sep 10 17:29:17.150448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.150419 2576 scope.go:117] "RemoveContainer" containerID="a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a" Sep 10 17:29:17.159164 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.159146 2576 scope.go:117] "RemoveContainer" containerID="1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002" Sep 10 17:29:17.159438 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:17.159420 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002\": container with ID starting with 1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002 not found: ID does not exist" containerID="1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002" Sep 10 17:29:17.159481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.159448 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002"} err="failed to get container status \"1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002\": rpc error: code = NotFound desc = could not find container \"1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002\": container with ID starting with 1bf4b941595056ff755d81ebad801ec930c624629defe0d2b99cc10e58b42002 not found: ID does not exist" Sep 10 17:29:17.159481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.159464 2576 scope.go:117] "RemoveContainer" containerID="a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a" Sep 10 17:29:17.159701 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:17.159681 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a\": container with ID starting with a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a not found: ID does not exist" containerID="a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a" Sep 10 17:29:17.159746 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.159707 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a"} err="failed to get container status \"a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a\": rpc error: code = NotFound desc = could not find container \"a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a\": container with ID starting with a9ceccac8c6f13858c320f1f033fe1b559742b5be27b7deec20e9f83825ee98a not found: ID does not exist" Sep 10 17:29:17.162101 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-envoy\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162168 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162116 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istiod-ca-cert\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162178 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-token\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-podinfo\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162246 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-certs\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162277 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-data\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162427 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162329 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpzps\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-kube-api-access-kpzps\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162427 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162399 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-socket\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162541 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-credential-socket\") pod \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\" (UID: \"1a5633e8-f77a-4313-a1f8-9fd99bf8b478\") " Sep 10 17:29:17.162594 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162560 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:29:17.162955 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162765 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:29:17.162955 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.162872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:29:17.163160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.163127 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:29:17.163160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.163148 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-data\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.163288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.163161 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:29:17.163288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.163169 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-credential-socket\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.163288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.163184 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istiod-ca-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.164673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.164646 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:29:17.164775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.164689 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:29:17.164775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.164704 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:29:17.164950 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.164932 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-kube-api-access-kpzps" (OuterVolumeSpecName: "kube-api-access-kpzps") pod "1a5633e8-f77a-4313-a1f8-9fd99bf8b478" (UID: "1a5633e8-f77a-4313-a1f8-9fd99bf8b478"). InnerVolumeSpecName "kube-api-access-kpzps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:29:17.263984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.263948 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzps\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-kube-api-access-kpzps\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.263984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.263980 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-socket\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.263984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.263992 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-envoy\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.264214 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.264000 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-token\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.264214 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.264008 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-istio-podinfo\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.264214 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.264016 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a5633e8-f77a-4313-a1f8-9fd99bf8b478-workload-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:17.467094 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.467058 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r"] Sep 10 17:29:17.474981 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:17.474954 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dc5a37-openshift-default-785c489f68-cdw2r"] Sep 10 17:29:18.487954 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:18.487924 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" path="/var/lib/kubelet/pods/1a5633e8-f77a-4313-a1f8-9fd99bf8b478/volumes" Sep 10 17:29:22.938353 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:22.938302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:23.568994 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:23.568949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:23.698464 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:23.698421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:23.769915 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:23.769861 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86"] Sep 10 17:29:23.770166 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:23.770138 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" podUID="b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" containerName="maas-api" containerID="cri-o://f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f" gracePeriod=30 Sep 10 17:29:23.809971 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:23.809866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:23.898753 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:23.898718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:23.979902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:23.979812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:24.039825 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.039792 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:29:24.067653 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.067347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:24.130952 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.130914 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-metrics-certs\") pod \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " Sep 10 17:29:24.131116 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.130986 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-openshift-service-ca\") pod \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " Sep 10 17:29:24.131116 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.131017 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpt52\" (UniqueName: \"kubernetes.io/projected/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-kube-api-access-bpt52\") pod \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " Sep 10 17:29:24.131205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.131127 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls\") pod \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\" (UID: \"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe\") " Sep 10 17:29:24.131438 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.131405 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" (UID: "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:29:24.131605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.131499 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:24.133381 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.133347 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-kube-api-access-bpt52" (OuterVolumeSpecName: "kube-api-access-bpt52") pod "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" (UID: "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe"). InnerVolumeSpecName "kube-api-access-bpt52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:29:24.133483 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.133351 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" (UID: "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:29:24.134147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.134121 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" (UID: "b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:29:24.139818 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.139742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:24.184892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.184850 2576 generic.go:356] "Generic (PLEG): container finished" podID="b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" containerID="f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f" exitCode=0 Sep 10 17:29:24.185056 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.184919 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" event={"ID":"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe","Type":"ContainerDied","Data":"f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f"} Sep 10 17:29:24.185056 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.184977 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" event={"ID":"b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe","Type":"ContainerDied","Data":"9b2ba5ca6a6ee50cc5b0e32af03508a791ad069e3f2b27ed28698fdcdac131ab"} Sep 10 17:29:24.185056 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.184999 2576 scope.go:117] "RemoveContainer" containerID="f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f" Sep 10 17:29:24.185056 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.184928 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86" Sep 10 17:29:24.194672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.194656 2576 scope.go:117] "RemoveContainer" containerID="f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f" Sep 10 17:29:24.195118 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:24.195083 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f\": container with ID starting with f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f not found: ID does not exist" containerID="f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f" Sep 10 17:29:24.195239 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.195126 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f"} err="failed to get container status \"f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f\": rpc error: code = NotFound desc = could not find container \"f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f\": container with ID starting with f39e0b9923b4c3615212a3e37efba402fd9daada1af2972f89280402b4ca581f not found: ID does not exist" Sep 10 17:29:24.215656 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.215633 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86"] Sep 10 17:29:24.219008 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.218983 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75bd9a-759f94b7fb-7kf86"] Sep 10 17:29:24.232865 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.232844 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:24.232865 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.232866 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpt52\" (UniqueName: \"kubernetes.io/projected/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-kube-api-access-bpt52\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:24.233021 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.232888 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:24.455629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.455539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:24.487189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.487148 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" path="/var/lib/kubelet/pods/b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe/volumes" Sep 10 17:29:24.974863 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:24.974829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:25.684771 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:25.684722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:25.864135 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:25.864081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:26.167687 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:26.167653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:29.095185 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:29.095094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:31.489979 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:31.489942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:31.605409 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:31.605371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:31.808053 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:31.807970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:32.407604 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.407566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:32.781503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781416 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7"] Sep 10 17:29:32.781854 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781819 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" Sep 10 17:29:32.781854 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781831 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" Sep 10 17:29:32.781854 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781843 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" containerName="maas-api" Sep 10 17:29:32.781854 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781848 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" containerName="maas-api" Sep 10 17:29:32.782015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781859 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" containerName="maas-api" Sep 10 17:29:32.782015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781866 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" containerName="maas-api" Sep 10 17:29:32.782015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781944 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3cf0b5-c7d6-4bb3-a1d9-0738fd5dbb4e" containerName="maas-api" Sep 10 17:29:32.782015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781954 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" Sep 10 17:29:32.782015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.781962 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0cfcd8a-9e20-4be6-a942-8f3cc2a5eafe" containerName="maas-api" Sep 10 17:29:32.785072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.785057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.788040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.788020 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-be583c-openshift-default-dockercfg-9c8pp\"" Sep 10 17:29:32.814088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.814061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7"] Sep 10 17:29:32.874272 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.874236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:32.920704 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-certs\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920704 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istiod-ca-cert\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-token\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-credential-socket\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-podinfo\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-socket\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngm9\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-kube-api-access-hngm9\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-envoy\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:32.920956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:32.920951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-data\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.021775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.021732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-socket\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.021982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.021782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngm9\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-kube-api-access-hngm9\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.021982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.021818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-envoy\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.021982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.021863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-data\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.021982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.021963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-certs\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.021982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.021982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istiod-ca-cert\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022285 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-token\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022285 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-credential-socket\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022285 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-podinfo\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022285 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-socket\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022510 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-certs\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022510 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-credential-socket\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022510 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-data\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.022976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.022952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istiod-ca-cert\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.024141 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.024120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-envoy\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.024582 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.024562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-podinfo\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.029847 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.029822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:33.032785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.032733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-token\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.033111 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.033086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngm9\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-kube-api-access-hngm9\") pod \"e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.095355 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.095319 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:33.394976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.394943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7"] Sep 10 17:29:33.396219 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:29:33.396182 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e68049f_cc72_4cd6_bb18_50bdf02c834c.slice/crio-61185d357d2c056898d7f9f0111d067aa4fac6b665e465c14bbcddf0ad3b8763 WatchSource:0}: Error finding container 61185d357d2c056898d7f9f0111d067aa4fac6b665e465c14bbcddf0ad3b8763: Status 404 returned error can't find the container with id 61185d357d2c056898d7f9f0111d067aa4fac6b665e465c14bbcddf0ad3b8763 Sep 10 17:29:33.398634 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.398597 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:33.398723 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.398669 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:33.398723 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:33.398699 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:34.055295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:34.055257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:34.140199 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:34.140161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:34.248536 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:34.248497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" event={"ID":"8e68049f-cc72-4cd6-bb18-50bdf02c834c","Type":"ContainerStarted","Data":"0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da"} Sep 10 17:29:34.248709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:34.248559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" event={"ID":"8e68049f-cc72-4cd6-bb18-50bdf02c834c","Type":"ContainerStarted","Data":"61185d357d2c056898d7f9f0111d067aa4fac6b665e465c14bbcddf0ad3b8763"} Sep 10 17:29:34.288446 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:34.288392 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podStartSLOduration=2.288374702 podStartE2EDuration="2.288374702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:34.286065066 +0000 UTC m=+1080.267145066" watchObservedRunningTime="2026-09-10 17:29:34.288374702 +0000 UTC m=+1080.269454700" Sep 10 17:29:35.096312 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:35.096275 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:35.102629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:35.102600 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:35.254975 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:35.254942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:35.256086 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:35.256059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:36.386637 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:36.386605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:36.495681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:36.495648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:37.643609 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.643581 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d"] Sep 10 17:29:37.644115 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.644028 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" Sep 10 17:29:37.644115 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.644042 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" Sep 10 17:29:37.644243 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.644132 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a5633e8-f77a-4313-a1f8-9fd99bf8b478" containerName="istio-proxy" Sep 10 17:29:37.649085 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.649061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.658708 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.658600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-be583c-serving-cert\"" Sep 10 17:29:37.670131 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.670008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d"] Sep 10 17:29:37.710514 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.710289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:37.789900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.785868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.789900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.786002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7420c73c-7447-4337-811d-54acd98bd47f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.789900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.786097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njf9n\" (UniqueName: \"kubernetes.io/projected/7420c73c-7447-4337-811d-54acd98bd47f-kube-api-access-njf9n\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.789900 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.786160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.823640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.823601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:37.887475 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.887434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7420c73c-7447-4337-811d-54acd98bd47f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.887640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.887546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njf9n\" (UniqueName: \"kubernetes.io/projected/7420c73c-7447-4337-811d-54acd98bd47f-kube-api-access-njf9n\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.887640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.887605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.887729 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.887667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.887808 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:37.887782 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-serving-cert: secret "maas-api-e2e-shared-a-w3-be583c-serving-cert" not found Sep 10 17:29:37.887890 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:37.887866 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls podName:7420c73c-7447-4337-811d-54acd98bd47f nodeName:}" failed. No retries permitted until 2026-09-10 17:29:38.387844932 +0000 UTC m=+1084.368924912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls") pod "maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" (UID: "7420c73c-7447-4337-811d-54acd98bd47f") : secret "maas-api-e2e-shared-a-w3-be583c-serving-cert" not found Sep 10 17:29:37.888187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.888163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7420c73c-7447-4337-811d-54acd98bd47f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.890615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.890588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:37.903945 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:37.903850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njf9n\" (UniqueName: \"kubernetes.io/projected/7420c73c-7447-4337-811d-54acd98bd47f-kube-api-access-njf9n\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:38.108613 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:38.108580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:38.393263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:38.393216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:38.395629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:38.395601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:38.577890 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:38.577831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:38.668348 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:38.668290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:38.780444 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:38.780399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:38.889302 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:38.888984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d"] Sep 10 17:29:38.904430 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:29:38.904336 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7420c73c_7447_4337_811d_54acd98bd47f.slice/crio-c638f97e93a4e90b0184866769d2fea0abb7b12aa341107a5020bea8aaaf293d WatchSource:0}: Error finding container c638f97e93a4e90b0184866769d2fea0abb7b12aa341107a5020bea8aaaf293d: Status 404 returned error can't find the container with id c638f97e93a4e90b0184866769d2fea0abb7b12aa341107a5020bea8aaaf293d Sep 10 17:29:39.288642 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:39.288527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" event={"ID":"7420c73c-7447-4337-811d-54acd98bd47f","Type":"ContainerStarted","Data":"c638f97e93a4e90b0184866769d2fea0abb7b12aa341107a5020bea8aaaf293d"} Sep 10 17:29:40.295668 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:40.295606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" event={"ID":"7420c73c-7447-4337-811d-54acd98bd47f","Type":"ContainerStarted","Data":"2e75660dfb945a865f1466e7200d410399eb0d32d05a3b95f9f35e26088cb316"} Sep 10 17:29:40.296106 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:40.295693 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:40.345428 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:40.345368 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" podStartSLOduration=3.046942564 podStartE2EDuration="3.345349777s" totalImagesPullingTime="298.407213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:40.342675272 +0000 UTC m=+1086.323755274" watchObservedRunningTime="2026-09-10 17:29:40.345349777 +0000 UTC m=+1086.326429774" Sep 10 17:29:44.679605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:44.679569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:45.501568 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.501520 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr"] Sep 10 17:29:45.502371 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.502339 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" podUID="2ae8107e-80d4-491a-b889-6c2fce25f10e" containerName="maas-api" containerID="cri-o://9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b" gracePeriod=30 Sep 10 17:29:45.548958 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.548926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:45.717904 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.702711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:45.813482 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.811967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:45.820940 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.820915 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:45.935074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.935031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:45.971867 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.971836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54857\" (UniqueName: \"kubernetes.io/projected/2ae8107e-80d4-491a-b889-6c2fce25f10e-kube-api-access-54857\") pod \"2ae8107e-80d4-491a-b889-6c2fce25f10e\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " Sep 10 17:29:45.972059 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.971951 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-metrics-certs\") pod \"2ae8107e-80d4-491a-b889-6c2fce25f10e\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " Sep 10 17:29:45.972125 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.972109 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls\") pod \"2ae8107e-80d4-491a-b889-6c2fce25f10e\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " Sep 10 17:29:45.972163 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.972140 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8107e-80d4-491a-b889-6c2fce25f10e-openshift-service-ca\") pod \"2ae8107e-80d4-491a-b889-6c2fce25f10e\" (UID: \"2ae8107e-80d4-491a-b889-6c2fce25f10e\") " Sep 10 17:29:45.972532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.972496 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae8107e-80d4-491a-b889-6c2fce25f10e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ae8107e-80d4-491a-b889-6c2fce25f10e" (UID: "2ae8107e-80d4-491a-b889-6c2fce25f10e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:29:45.974085 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.974050 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae8107e-80d4-491a-b889-6c2fce25f10e-kube-api-access-54857" (OuterVolumeSpecName: "kube-api-access-54857") pod "2ae8107e-80d4-491a-b889-6c2fce25f10e" (UID: "2ae8107e-80d4-491a-b889-6c2fce25f10e"). InnerVolumeSpecName "kube-api-access-54857". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:29:45.974175 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.974112 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ae8107e-80d4-491a-b889-6c2fce25f10e" (UID: "2ae8107e-80d4-491a-b889-6c2fce25f10e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:29:45.974217 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:45.974178 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ae8107e-80d4-491a-b889-6c2fce25f10e" (UID: "2ae8107e-80d4-491a-b889-6c2fce25f10e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:29:46.026548 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.026515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:46.072970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.072941 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:46.072970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.072974 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8107e-80d4-491a-b889-6c2fce25f10e-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:46.073154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.072983 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54857\" (UniqueName: \"kubernetes.io/projected/2ae8107e-80d4-491a-b889-6c2fce25f10e-kube-api-access-54857\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:46.073154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.072992 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae8107e-80d4-491a-b889-6c2fce25f10e-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:29:46.313671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.313635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:29:46.335552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.335464 2576 generic.go:356] "Generic (PLEG): container finished" podID="2ae8107e-80d4-491a-b889-6c2fce25f10e" containerID="9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b" exitCode=0 Sep 10 17:29:46.335552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.335515 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" Sep 10 17:29:46.335783 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.335549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" event={"ID":"2ae8107e-80d4-491a-b889-6c2fce25f10e","Type":"ContainerDied","Data":"9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b"} Sep 10 17:29:46.335783 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.335599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr" event={"ID":"2ae8107e-80d4-491a-b889-6c2fce25f10e","Type":"ContainerDied","Data":"9f3e396b30dd866c479b865354339b14ef3b1b9ea27e5e7021eaff2d23c03f13"} Sep 10 17:29:46.335783 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.335624 2576 scope.go:117] "RemoveContainer" containerID="9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b" Sep 10 17:29:46.345992 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.345969 2576 scope.go:117] "RemoveContainer" containerID="9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b" Sep 10 17:29:46.346296 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:29:46.346277 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b\": container with ID starting with 9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b not found: ID does not exist" containerID="9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b" Sep 10 17:29:46.346356 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.346305 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b"} err="failed to get container status \"9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b\": rpc error: code = NotFound desc = could not find container \"9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b\": container with ID starting with 9c7e6e140cce95bdc7f4846703d4d97d16978dda1e070b99710d11c2d6a4626b not found: ID does not exist" Sep 10 17:29:46.366211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.366179 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr"] Sep 10 17:29:46.376947 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.376916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:46.376947 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.376935 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6dc6af55-68cbbb8865-vvprr"] Sep 10 17:29:46.486841 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:46.486805 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae8107e-80d4-491a-b889-6c2fce25f10e" path="/var/lib/kubelet/pods/2ae8107e-80d4-491a-b889-6c2fce25f10e/volumes" Sep 10 17:29:51.298078 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:51.298035 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 17:29:51.298505 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:51.298104 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 17:29:52.859467 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:52.859393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:52.960434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:52.960399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:53.175088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.174992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:53.383298 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.383266 2576 generic.go:356] "Generic (PLEG): container finished" podID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerID="0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da" exitCode=0 Sep 10 17:29:53.383488 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.383346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" event={"ID":"8e68049f-cc72-4cd6-bb18-50bdf02c834c","Type":"ContainerDied","Data":"0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da"} Sep 10 17:29:53.383774 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.383756 2576 scope.go:117] "RemoveContainer" containerID="0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da" Sep 10 17:29:53.384843 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.384805 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:53.384988 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.384903 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:53.384988 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.384949 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:53.546664 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.546619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:53.673493 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.673459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:53.770497 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:53.770413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:54.126530 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:54.126496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:29:54.393978 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:54.393860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" event={"ID":"8e68049f-cc72-4cd6-bb18-50bdf02c834c","Type":"ContainerStarted","Data":"addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4"} Sep 10 17:29:55.096639 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:55.096597 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:29:55.096852 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:55.096732 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 17:29:55.096852 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:55.096801 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 17:29:56.096426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:56.096380 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 17:29:56.096950 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:56.096451 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 17:29:57.095826 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:57.095775 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 17:29:57.096071 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:57.095853 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 17:29:58.096718 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:58.096672 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 17:29:58.097250 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:58.096741 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 17:29:59.096082 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:59.096034 2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 10 17:29:59.096400 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:59.096367 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 10 17:29:59.417325 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:29:59.417233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:00.258270 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:00.258241 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:30:00.258553 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:00.258530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:30:00.259378 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:00.259360 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:30:01.177836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:01.177789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:01.335805 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:01.335765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:01.466436 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:01.466265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:01.591790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:01.591753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:01.899579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:01.899544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:02.217780 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:02.217688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:03.339389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:03.339356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:05.280933 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:05.280897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:05.465389 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:05.465353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:06.506830 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.506790 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w"] Sep 10 17:30:06.507332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.507239 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae8107e-80d4-491a-b889-6c2fce25f10e" containerName="maas-api" Sep 10 17:30:06.507332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.507254 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae8107e-80d4-491a-b889-6c2fce25f10e" containerName="maas-api" Sep 10 17:30:06.507332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.507330 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae8107e-80d4-491a-b889-6c2fce25f10e" containerName="maas-api" Sep 10 17:30:06.510694 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.510673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.514224 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.514185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6806f1-serving-cert\"" Sep 10 17:30:06.543346 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.543311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w"] Sep 10 17:30:06.593531 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.593494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rtpg\" (UniqueName: \"kubernetes.io/projected/a56f58e1-d3da-4b2b-a691-810c3f546eb9-kube-api-access-2rtpg\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.593710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.593538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56f58e1-d3da-4b2b-a691-810c3f546eb9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.593710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.593621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.593710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.593657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.654800 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.654766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:06.695259 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.695217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rtpg\" (UniqueName: \"kubernetes.io/projected/a56f58e1-d3da-4b2b-a691-810c3f546eb9-kube-api-access-2rtpg\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.695259 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.695267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56f58e1-d3da-4b2b-a691-810c3f546eb9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.695500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.695308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.695500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.695335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.695500 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:30:06.695445 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-serving-cert: secret "maas-api-e2e-shared-b-w3-6806f1-serving-cert" not found Sep 10 17:30:06.695608 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:30:06.695529 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls podName:a56f58e1-d3da-4b2b-a691-810c3f546eb9 nodeName:}" failed. No retries permitted until 2026-09-10 17:30:07.195507652 +0000 UTC m=+1113.176587644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls") pod "maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" (UID: "a56f58e1-d3da-4b2b-a691-810c3f546eb9") : secret "maas-api-e2e-shared-b-w3-6806f1-serving-cert" not found Sep 10 17:30:06.695975 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.695944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56f58e1-d3da-4b2b-a691-810c3f546eb9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.697838 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.697802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.713352 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.713322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rtpg\" (UniqueName: \"kubernetes.io/projected/a56f58e1-d3da-4b2b-a691-810c3f546eb9-kube-api-access-2rtpg\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:06.772976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:06.772861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:07.201201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:07.201156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:07.201421 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:30:07.201340 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-serving-cert: secret "maas-api-e2e-shared-b-w3-6806f1-serving-cert" not found Sep 10 17:30:07.201421 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:30:07.201418 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls podName:a56f58e1-d3da-4b2b-a691-810c3f546eb9 nodeName:}" failed. No retries permitted until 2026-09-10 17:30:08.201397627 +0000 UTC m=+1114.182477617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls") pod "maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" (UID: "a56f58e1-d3da-4b2b-a691-810c3f546eb9") : secret "maas-api-e2e-shared-b-w3-6806f1-serving-cert" not found Sep 10 17:30:07.409446 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:07.409405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:07.533962 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:07.533846 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:07.930855 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:07.930821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:08.211186 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:08.211097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:08.213654 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:08.213630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:08.324064 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:08.324028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:08.592234 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:08.592197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:08.606790 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:08.606761 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w"] Sep 10 17:30:08.608595 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:30:08.608574 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56f58e1_d3da_4b2b_a691_810c3f546eb9.slice/crio-22488bea169b6876626c744c97fd498db13a44e631c5b6f34c5e6620a10e8dda WatchSource:0}: Error finding container 22488bea169b6876626c744c97fd498db13a44e631c5b6f34c5e6620a10e8dda: Status 404 returned error can't find the container with id 22488bea169b6876626c744c97fd498db13a44e631c5b6f34c5e6620a10e8dda Sep 10 17:30:09.488926 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:09.488863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" event={"ID":"a56f58e1-d3da-4b2b-a691-810c3f546eb9","Type":"ContainerStarted","Data":"df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21"} Sep 10 17:30:09.489103 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:09.488944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" event={"ID":"a56f58e1-d3da-4b2b-a691-810c3f546eb9","Type":"ContainerStarted","Data":"22488bea169b6876626c744c97fd498db13a44e631c5b6f34c5e6620a10e8dda"} Sep 10 17:30:09.489103 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:09.489008 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:09.514498 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:09.514443 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" podStartSLOduration=3.160994677 podStartE2EDuration="3.514425587s" totalImagesPullingTime="353.43091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:09.513427318 +0000 UTC m=+1115.494507316" watchObservedRunningTime="2026-09-10 17:30:09.514425587 +0000 UTC m=+1115.495505586" Sep 10 17:30:09.796908 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:09.796806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:09.894716 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:09.894675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:10.744372 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:10.744337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:11.090622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:11.090579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:11.745267 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:11.745231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:14.366160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:14.366127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:14.488689 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:14.488654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:15.501128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.501100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:30:15.568780 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.568743 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2"] Sep 10 17:30:15.573260 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.573235 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.578184 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.578158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-33b7bb-serving-cert\"" Sep 10 17:30:15.590053 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.590015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2"] Sep 10 17:30:15.616506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.616473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:15.690387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.690354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44dc\" (UniqueName: \"kubernetes.io/projected/3aa57572-1acd-4bd2-a75e-d9613e3526b6-kube-api-access-l44dc\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.690601 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.690417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.690601 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.690504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57572-1acd-4bd2-a75e-d9613e3526b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.690601 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.690544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.720140 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.720106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:15.791287 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.791192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44dc\" (UniqueName: \"kubernetes.io/projected/3aa57572-1acd-4bd2-a75e-d9613e3526b6-kube-api-access-l44dc\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.791287 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.791255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.791532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.791314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57572-1acd-4bd2-a75e-d9613e3526b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.791532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.791348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.792036 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.792008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57572-1acd-4bd2-a75e-d9613e3526b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.793863 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.793836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.793983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.793873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.800340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.800320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44dc\" (UniqueName: \"kubernetes.io/projected/3aa57572-1acd-4bd2-a75e-d9613e3526b6-kube-api-access-l44dc\") pod \"maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:15.818859 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.818825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:15.887107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:15.887072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:16.163555 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:16.163521 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2"] Sep 10 17:30:16.165170 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:30:16.165139 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa57572_1acd_4bd2_a75e_d9613e3526b6.slice/crio-9ffe1840cec349ee3657d0402911c682b7efb35bf40c4974c10b4e750f7736b6 WatchSource:0}: Error finding container 9ffe1840cec349ee3657d0402911c682b7efb35bf40c4974c10b4e750f7736b6: Status 404 returned error can't find the container with id 9ffe1840cec349ee3657d0402911c682b7efb35bf40c4974c10b4e750f7736b6 Sep 10 17:30:16.260455 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:16.260415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:16.349896 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:16.349857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:16.534482 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:16.534439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" event={"ID":"3aa57572-1acd-4bd2-a75e-d9613e3526b6","Type":"ContainerStarted","Data":"fd8d097c51570323ff73ceccb692801c5abdf71d5f178574a9ec26d8be9bfc0b"} Sep 10 17:30:16.535021 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:16.534997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" event={"ID":"3aa57572-1acd-4bd2-a75e-d9613e3526b6","Type":"ContainerStarted","Data":"9ffe1840cec349ee3657d0402911c682b7efb35bf40c4974c10b4e750f7736b6"} Sep 10 17:30:16.535240 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:16.535222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:16.564842 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:16.564787 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" podStartSLOduration=1.275220718 podStartE2EDuration="1.564770541s" totalImagesPullingTime="289.549823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:16.562579962 +0000 UTC m=+1122.543659961" watchObservedRunningTime="2026-09-10 17:30:16.564770541 +0000 UTC m=+1122.545850539" Sep 10 17:30:23.553549 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:23.553518 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:30:28.017390 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:28.017351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:30.331636 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:30.331597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:36.935426 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:36.935383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:38.630899 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:38.630847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:39.327823 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:39.327786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:39.683576 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:39.683488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:40.097296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:40.097255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:40.238922 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:40.238871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:43.678243 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:43.678207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:43.807607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:43.807564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:45.007286 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.007253 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9"] Sep 10 17:30:45.012301 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.012279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.022985 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.022958 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-edbcb5-serving-cert\"" Sep 10 17:30:45.043313 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.043285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9"] Sep 10 17:30:45.063319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.063291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.063481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.063371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9vx\" (UniqueName: \"kubernetes.io/projected/21b44521-ebe9-43ff-be02-017de75aabcf-kube-api-access-2q9vx\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.063481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.063402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.063481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.063435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b44521-ebe9-43ff-be02-017de75aabcf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.096687 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.096647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:45.164761 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.164724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9vx\" (UniqueName: \"kubernetes.io/projected/21b44521-ebe9-43ff-be02-017de75aabcf-kube-api-access-2q9vx\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.164970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.164769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.164970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.164800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b44521-ebe9-43ff-be02-017de75aabcf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.164970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.164848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.165096 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:30:45.165023 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-serving-cert: secret "maas-api-e2e-shared-b-w2-edbcb5-serving-cert" not found Sep 10 17:30:45.165138 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:30:45.165097 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls podName:21b44521-ebe9-43ff-be02-017de75aabcf nodeName:}" failed. No retries permitted until 2026-09-10 17:30:45.665076127 +0000 UTC m=+1151.646156110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls") pod "maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" (UID: "21b44521-ebe9-43ff-be02-017de75aabcf") : secret "maas-api-e2e-shared-b-w2-edbcb5-serving-cert" not found Sep 10 17:30:45.165620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.165594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b44521-ebe9-43ff-be02-017de75aabcf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.167415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.167394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.184360 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.184329 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9vx\" (UniqueName: \"kubernetes.io/projected/21b44521-ebe9-43ff-be02-017de75aabcf-kube-api-access-2q9vx\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.217487 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.217455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:45.349147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.349120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:45.670784 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.670688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.673198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.673179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.840260 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.840222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:45.922476 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.922386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:45.968213 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:45.968183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:46.208899 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:46.208857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9"] Sep 10 17:30:46.210434 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:30:46.210405 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b44521_ebe9_43ff_be02_017de75aabcf.slice/crio-771f06682999dde8caa66618529a651575240e86cc0cf1e810476bf753adec1f WatchSource:0}: Error finding container 771f06682999dde8caa66618529a651575240e86cc0cf1e810476bf753adec1f: Status 404 returned error can't find the container with id 771f06682999dde8caa66618529a651575240e86cc0cf1e810476bf753adec1f Sep 10 17:30:46.710637 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:46.710603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" event={"ID":"21b44521-ebe9-43ff-be02-017de75aabcf","Type":"ContainerStarted","Data":"db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793"} Sep 10 17:30:46.710827 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:46.710660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" event={"ID":"21b44521-ebe9-43ff-be02-017de75aabcf","Type":"ContainerStarted","Data":"771f06682999dde8caa66618529a651575240e86cc0cf1e810476bf753adec1f"} Sep 10 17:30:46.710827 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:46.710691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:46.747100 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:46.747047 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" podStartSLOduration=2.371936636 podStartE2EDuration="2.747030125s" totalImagesPullingTime="375.093489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:46.744180424 +0000 UTC m=+1152.725260421" watchObservedRunningTime="2026-09-10 17:30:46.747030125 +0000 UTC m=+1152.728110123" Sep 10 17:30:52.721621 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:52.721592 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:30:53.691189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:53.691156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:30:53.775280 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:30:53.775245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:08.597920 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:08.597815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:12.581351 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:12.581314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:18.020373 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:18.020333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:18.512234 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:18.512198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:18.609461 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:18.609426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:21.649680 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:21.649619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:21.760449 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:21.760416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:22.802857 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.802824 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl"] Sep 10 17:31:22.807928 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.807907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.810917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.810897 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9d28f397-serving-cert\"" Sep 10 17:31:22.822811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.822782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl"] Sep 10 17:31:22.835615 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.835580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:22.846605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.846577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.846749 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.846693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dbcc03-cbf4-4213-855f-d98bccfc5405-openshift-service-ca\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.846821 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.846780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nh9n\" (UniqueName: \"kubernetes.io/projected/36dbcc03-cbf4-4213-855f-d98bccfc5405-kube-api-access-7nh9n\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.846874 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.846828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-metrics-certs\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.913492 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.913454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:22.954667 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.954631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dbcc03-cbf4-4213-855f-d98bccfc5405-openshift-service-ca\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.954853 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.954717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nh9n\" (UniqueName: \"kubernetes.io/projected/36dbcc03-cbf4-4213-855f-d98bccfc5405-kube-api-access-7nh9n\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.954853 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.954748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-metrics-certs\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.954853 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.954819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.955080 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:31:22.954946 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-serving-cert: secret "maas-api-e2e-mt-9d28f397-serving-cert" not found Sep 10 17:31:22.955080 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:31:22.955012 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls podName:36dbcc03-cbf4-4213-855f-d98bccfc5405 nodeName:}" failed. No retries permitted until 2026-09-10 17:31:23.454993383 +0000 UTC m=+1189.436073359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls") pod "maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" (UID: "36dbcc03-cbf4-4213-855f-d98bccfc5405") : secret "maas-api-e2e-mt-9d28f397-serving-cert" not found Sep 10 17:31:22.955466 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.955435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dbcc03-cbf4-4213-855f-d98bccfc5405-openshift-service-ca\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.957237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.957214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-metrics-certs\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:22.969102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:22.969077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nh9n\" (UniqueName: \"kubernetes.io/projected/36dbcc03-cbf4-4213-855f-d98bccfc5405-kube-api-access-7nh9n\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:23.048267 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:23.048232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:23.425747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:23.425707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:23.460575 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:23.460546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:23.463016 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:23.462991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls\") pod \"maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:23.515890 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:23.515850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:23.718816 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:23.718733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:23.997434 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:23.997405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl"] Sep 10 17:31:23.998659 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:31:23.998630 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dbcc03_cbf4_4213_855f_d98bccfc5405.slice/crio-ac36d9b28d9b5aca3c6e163c01f21bef4c783fe10b656f60250ebe267c8ae442 WatchSource:0}: Error finding container ac36d9b28d9b5aca3c6e163c01f21bef4c783fe10b656f60250ebe267c8ae442: Status 404 returned error can't find the container with id ac36d9b28d9b5aca3c6e163c01f21bef4c783fe10b656f60250ebe267c8ae442 Sep 10 17:31:24.930821 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:24.930781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" event={"ID":"36dbcc03-cbf4-4213-855f-d98bccfc5405","Type":"ContainerStarted","Data":"b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9"} Sep 10 17:31:24.931045 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:24.930846 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" event={"ID":"36dbcc03-cbf4-4213-855f-d98bccfc5405","Type":"ContainerStarted","Data":"ac36d9b28d9b5aca3c6e163c01f21bef4c783fe10b656f60250ebe267c8ae442"} Sep 10 17:31:24.931045 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:24.931011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:24.955585 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:24.955534 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" podStartSLOduration=2.561093459 podStartE2EDuration="2.955521176s" totalImagesPullingTime="394.427717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:24.952187501 +0000 UTC m=+1190.933267500" watchObservedRunningTime="2026-09-10 17:31:24.955521176 +0000 UTC m=+1190.936601175" Sep 10 17:31:30.942966 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:30.942936 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:31:34.531365 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:34.531329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:31:34.539079 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:34.539047 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:31:46.593431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:46.593387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:49.359579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:49.359543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:31:58.609659 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:31:58.609621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:07.037489 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:07.037451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:36.521046 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:36.520965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:52.091835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.091801 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl"] Sep 10 17:32:52.092314 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.092068 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" podUID="36dbcc03-cbf4-4213-855f-d98bccfc5405" containerName="maas-api" containerID="cri-o://b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9" gracePeriod=30 Sep 10 17:32:52.127275 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.127231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:52.222031 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.221992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:52.321017 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.320987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:52.357721 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.357662 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:32:52.418819 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.418784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:52.438577 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.438542 2576 generic.go:356] "Generic (PLEG): container finished" podID="36dbcc03-cbf4-4213-855f-d98bccfc5405" containerID="b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9" exitCode=0 Sep 10 17:32:52.438733 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.438588 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" Sep 10 17:32:52.438733 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.438625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" event={"ID":"36dbcc03-cbf4-4213-855f-d98bccfc5405","Type":"ContainerDied","Data":"b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9"} Sep 10 17:32:52.438733 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.438678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl" event={"ID":"36dbcc03-cbf4-4213-855f-d98bccfc5405","Type":"ContainerDied","Data":"ac36d9b28d9b5aca3c6e163c01f21bef4c783fe10b656f60250ebe267c8ae442"} Sep 10 17:32:52.438733 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.438703 2576 scope.go:117] "RemoveContainer" containerID="b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9" Sep 10 17:32:52.448349 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.448331 2576 scope.go:117] "RemoveContainer" containerID="b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9" Sep 10 17:32:52.448638 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:32:52.448616 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9\": container with ID starting with b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9 not found: ID does not exist" containerID="b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9" Sep 10 17:32:52.448683 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.448649 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9"} err="failed to get container status \"b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9\": rpc error: code = NotFound desc = could not find container \"b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9\": container with ID starting with b126902e5489ecf6aa7468ea095f68feb0453498d91c2757807a58035db70ea9 not found: ID does not exist" Sep 10 17:32:52.507296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.507261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:52.525333 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.525304 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dbcc03-cbf4-4213-855f-d98bccfc5405-openshift-service-ca\") pod \"36dbcc03-cbf4-4213-855f-d98bccfc5405\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " Sep 10 17:32:52.525473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.525365 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls\") pod \"36dbcc03-cbf4-4213-855f-d98bccfc5405\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " Sep 10 17:32:52.525473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.525407 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-metrics-certs\") pod \"36dbcc03-cbf4-4213-855f-d98bccfc5405\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " Sep 10 17:32:52.525473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.525459 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nh9n\" (UniqueName: \"kubernetes.io/projected/36dbcc03-cbf4-4213-855f-d98bccfc5405-kube-api-access-7nh9n\") pod \"36dbcc03-cbf4-4213-855f-d98bccfc5405\" (UID: \"36dbcc03-cbf4-4213-855f-d98bccfc5405\") " Sep 10 17:32:52.525767 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.525739 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dbcc03-cbf4-4213-855f-d98bccfc5405-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36dbcc03-cbf4-4213-855f-d98bccfc5405" (UID: "36dbcc03-cbf4-4213-855f-d98bccfc5405"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:32:52.527938 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.527863 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dbcc03-cbf4-4213-855f-d98bccfc5405-kube-api-access-7nh9n" (OuterVolumeSpecName: "kube-api-access-7nh9n") pod "36dbcc03-cbf4-4213-855f-d98bccfc5405" (UID: "36dbcc03-cbf4-4213-855f-d98bccfc5405"). InnerVolumeSpecName "kube-api-access-7nh9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:32:52.528049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.528017 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36dbcc03-cbf4-4213-855f-d98bccfc5405" (UID: "36dbcc03-cbf4-4213-855f-d98bccfc5405"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:32:52.528049 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.528034 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36dbcc03-cbf4-4213-855f-d98bccfc5405" (UID: "36dbcc03-cbf4-4213-855f-d98bccfc5405"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:32:52.626643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.626542 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dbcc03-cbf4-4213-855f-d98bccfc5405-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:32:52.626643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.626579 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:32:52.626643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.626589 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dbcc03-cbf4-4213-855f-d98bccfc5405-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:32:52.626643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.626603 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nh9n\" (UniqueName: \"kubernetes.io/projected/36dbcc03-cbf4-4213-855f-d98bccfc5405-kube-api-access-7nh9n\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:32:52.763848 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.763809 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl"] Sep 10 17:32:52.767988 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:52.767961 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d28f397-56d9bd8847-tt8sl"] Sep 10 17:32:54.486850 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:54.486815 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36dbcc03-cbf4-4213-855f-d98bccfc5405" path="/var/lib/kubelet/pods/36dbcc03-cbf4-4213-855f-d98bccfc5405/volumes" Sep 10 17:32:55.679766 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:55.679737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:32:59.464976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:32:59.464936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:33:00.831173 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:33:00.831131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:33:36.570373 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:33:36.570338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:33:37.761657 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:33:37.761603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:33:38.547620 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:33:38.547580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:33:38.664028 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:33:38.663992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:33:44.410842 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:33:44.410803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:03.025313 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:03.025207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:09.511387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:09.511351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:09.608970 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:09.608926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:09.763483 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:09.763392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:11.083210 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:11.083174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:11.187029 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:11.186994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:11.301998 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:11.301946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:11.910206 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:11.910171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:12.022084 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:12.022045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:13.386876 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:13.386841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:19.562225 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:19.562183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:19.656715 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:19.656672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:20.271271 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:20.271236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:25.791237 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:25.791192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:27.240669 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:27.240631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:27.349648 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:27.349610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:27.471551 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:27.471516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:37.984994 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:37.984959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:38.229693 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:38.229656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:54.973505 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:54.973463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:34:59.235573 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:34:59.235539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:00.325137 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:00.325100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:04.446549 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:04.446508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:05.958209 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:05.958173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:06.140892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:06.140838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:06.380078 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:06.380030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:06.685076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:06.684964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:06.787778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:06.787745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:09.924555 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:09.924517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:10.066798 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:10.066769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:11.210257 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.210227 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9"] Sep 10 17:35:11.210672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.210658 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dbcc03-cbf4-4213-855f-d98bccfc5405" containerName="maas-api" Sep 10 17:35:11.210717 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.210674 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dbcc03-cbf4-4213-855f-d98bccfc5405" containerName="maas-api" Sep 10 17:35:11.211083 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.210748 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="36dbcc03-cbf4-4213-855f-d98bccfc5405" containerName="maas-api" Sep 10 17:35:11.213777 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.213760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.216860 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.216840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2549d0-serving-cert\"" Sep 10 17:35:11.227612 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.227589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9"] Sep 10 17:35:11.240189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.240144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:11.288849 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.288816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nzm\" (UniqueName: \"kubernetes.io/projected/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-kube-api-access-65nzm\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.289032 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.288864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.289032 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.288990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-metrics-certs\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.289130 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.289050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-openshift-service-ca\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.338033 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.338001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:11.390314 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.390277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nzm\" (UniqueName: \"kubernetes.io/projected/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-kube-api-access-65nzm\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.390511 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.390320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.390511 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.390370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-metrics-certs\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.390511 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.390414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-openshift-service-ca\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.390683 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:35:11.390505 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-serving-cert: secret "maas-api-e2e-api-a-2549d0-serving-cert" not found Sep 10 17:35:11.390683 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:35:11.390604 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls podName:5e6ffe65-3463-40a2-abd5-cf7ee05b2488 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:11.890584731 +0000 UTC m=+1417.871664721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls") pod "maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" (UID: "5e6ffe65-3463-40a2-abd5-cf7ee05b2488") : secret "maas-api-e2e-api-a-2549d0-serving-cert" not found Sep 10 17:35:11.391144 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.391127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-openshift-service-ca\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.392822 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.392804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-metrics-certs\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.419674 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.419641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nzm\" (UniqueName: \"kubernetes.io/projected/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-kube-api-access-65nzm\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.506841 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.506754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:11.903514 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.903479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.906174 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.906150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls\") pod \"maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:11.990782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:11.990733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:12.123593 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:12.123558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:12.135266 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:12.135238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:12.418561 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:12.418532 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9"] Sep 10 17:35:12.419625 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:35:12.419599 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6ffe65_3463_40a2_abd5_cf7ee05b2488.slice/crio-aeb58e2a8cb43ad14c6c5a312df97881c77eadd0c7bf532fc330dee4ada96f2f WatchSource:0}: Error finding container aeb58e2a8cb43ad14c6c5a312df97881c77eadd0c7bf532fc330dee4ada96f2f: Status 404 returned error can't find the container with id aeb58e2a8cb43ad14c6c5a312df97881c77eadd0c7bf532fc330dee4ada96f2f Sep 10 17:35:12.421440 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:12.421418 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:35:13.267415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:13.267335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" event={"ID":"5e6ffe65-3463-40a2-abd5-cf7ee05b2488","Type":"ContainerStarted","Data":"6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff"} Sep 10 17:35:13.267415 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:13.267387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" event={"ID":"5e6ffe65-3463-40a2-abd5-cf7ee05b2488","Type":"ContainerStarted","Data":"aeb58e2a8cb43ad14c6c5a312df97881c77eadd0c7bf532fc330dee4ada96f2f"} Sep 10 17:35:13.267614 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:13.267460 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:13.292814 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:13.292769 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" podStartSLOduration=1.887393654 podStartE2EDuration="2.292753691s" totalImagesPullingTime="405.360037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:13.289695637 +0000 UTC m=+1419.270775637" watchObservedRunningTime="2026-09-10 17:35:13.292753691 +0000 UTC m=+1419.273833689" Sep 10 17:35:13.989875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:13.989821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:15.305644 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:15.305608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:19.277857 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:19.277827 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:35:28.935774 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:28.935689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:32.132015 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:32.131979 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:34.502333 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:34.502298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:37.980627 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:37.980580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:39.971411 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:39.971374 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:40.430729 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:40.430696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:40.757951 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:40.757845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:40.849413 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:40.849371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:44.160682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:44.160630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:44.262597 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:44.262568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:45.322423 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.322391 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc"] Sep 10 17:35:45.326484 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.326461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.329318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.329292 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-65fac0-serving-cert\"" Sep 10 17:35:45.340977 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.340955 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc"] Sep 10 17:35:45.365269 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.365233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:45.447933 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.447898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a05ce-aefd-4594-9f89-cca90a842229-openshift-service-ca\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.448113 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.447980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvlz\" (UniqueName: \"kubernetes.io/projected/058a05ce-aefd-4594-9f89-cca90a842229-kube-api-access-fbvlz\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.448113 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.448080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.448113 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.448097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-metrics-certs\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.462712 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.462680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:45.549457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.549413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.549457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.549458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-metrics-certs\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.549728 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.549535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a05ce-aefd-4594-9f89-cca90a842229-openshift-service-ca\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.549728 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:35:45.549566 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-serving-cert: secret "maas-api-e2e-api-b-65fac0-serving-cert" not found Sep 10 17:35:45.549728 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.549604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvlz\" (UniqueName: \"kubernetes.io/projected/058a05ce-aefd-4594-9f89-cca90a842229-kube-api-access-fbvlz\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.549728 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:35:45.549649 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls podName:058a05ce-aefd-4594-9f89-cca90a842229 nodeName:}" failed. No retries permitted until 2026-09-10 17:35:46.049628156 +0000 UTC m=+1452.030708149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls") pod "maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" (UID: "058a05ce-aefd-4594-9f89-cca90a842229") : secret "maas-api-e2e-api-b-65fac0-serving-cert" not found Sep 10 17:35:45.550311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.550286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a05ce-aefd-4594-9f89-cca90a842229-openshift-service-ca\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.552098 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.552070 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-metrics-certs\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.558573 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.558553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvlz\" (UniqueName: \"kubernetes.io/projected/058a05ce-aefd-4594-9f89-cca90a842229-kube-api-access-fbvlz\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:45.794869 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.794776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:45.983728 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:45.983694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:46.055804 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:46.055720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:46.058246 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:46.058226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls\") pod \"maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:46.209383 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:46.209338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:35:46.238012 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:46.237979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:46.524722 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:46.524665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc"] Sep 10 17:35:46.527035 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:35:46.527007 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058a05ce_aefd_4594_9f89_cca90a842229.slice/crio-d1ed43d12c6e38bdf53906169d32ca10bcfccc90b5499b027072b4284a51f37d WatchSource:0}: Error finding container d1ed43d12c6e38bdf53906169d32ca10bcfccc90b5499b027072b4284a51f37d: Status 404 returned error can't find the container with id d1ed43d12c6e38bdf53906169d32ca10bcfccc90b5499b027072b4284a51f37d Sep 10 17:35:47.465334 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:47.465296 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" event={"ID":"058a05ce-aefd-4594-9f89-cca90a842229","Type":"ContainerStarted","Data":"483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe"} Sep 10 17:35:47.465539 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:47.465519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:35:47.465590 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:47.465562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" event={"ID":"058a05ce-aefd-4594-9f89-cca90a842229","Type":"ContainerStarted","Data":"d1ed43d12c6e38bdf53906169d32ca10bcfccc90b5499b027072b4284a51f37d"} Sep 10 17:35:47.488983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:47.488933 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" podStartSLOduration=2.135278998 podStartE2EDuration="2.488918591s" totalImagesPullingTime="353.639593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:47.486435307 +0000 UTC m=+1453.467515306" watchObservedRunningTime="2026-09-10 17:35:47.488918591 +0000 UTC m=+1453.469998586" Sep 10 17:35:53.476917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:35:53.476868 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:36:07.311991 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:07.311953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:08.290920 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:08.290871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:09.017917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:09.017859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:11.989695 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:11.989649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:18.944176 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:18.944138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:27.432833 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.432797 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9"] Sep 10 17:36:27.433308 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.433081 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" podUID="5e6ffe65-3463-40a2-abd5-cf7ee05b2488" containerName="maas-api" containerID="cri-o://6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff" gracePeriod=30 Sep 10 17:36:27.458510 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.458476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:27.556294 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.556265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:27.655532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.655495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:27.694587 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.694522 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:36:27.704843 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.704808 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e6ffe65-3463-40a2-abd5-cf7ee05b2488" containerID="6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff" exitCode=0 Sep 10 17:36:27.705052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.704853 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" Sep 10 17:36:27.705052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.704861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" event={"ID":"5e6ffe65-3463-40a2-abd5-cf7ee05b2488","Type":"ContainerDied","Data":"6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff"} Sep 10 17:36:27.705052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.704940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9" event={"ID":"5e6ffe65-3463-40a2-abd5-cf7ee05b2488","Type":"ContainerDied","Data":"aeb58e2a8cb43ad14c6c5a312df97881c77eadd0c7bf532fc330dee4ada96f2f"} Sep 10 17:36:27.705052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.704998 2576 scope.go:117] "RemoveContainer" containerID="6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff" Sep 10 17:36:27.715736 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.715694 2576 scope.go:117] "RemoveContainer" containerID="6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff" Sep 10 17:36:27.716187 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:36:27.716131 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff\": container with ID starting with 6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff not found: ID does not exist" containerID="6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff" Sep 10 17:36:27.716187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.716173 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff"} err="failed to get container status \"6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff\": rpc error: code = NotFound desc = could not find container \"6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff\": container with ID starting with 6ff33c17f22690015ed6af381930062b24f38844a5337249b4cc4beabebb76ff not found: ID does not exist" Sep 10 17:36:27.725858 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.725825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:27.783021 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.782993 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65nzm\" (UniqueName: \"kubernetes.io/projected/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-kube-api-access-65nzm\") pod \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " Sep 10 17:36:27.783191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.783046 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-metrics-certs\") pod \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " Sep 10 17:36:27.783191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.783080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls\") pod \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " Sep 10 17:36:27.783191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.783120 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-openshift-service-ca\") pod \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\" (UID: \"5e6ffe65-3463-40a2-abd5-cf7ee05b2488\") " Sep 10 17:36:27.783564 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.783537 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e6ffe65-3463-40a2-abd5-cf7ee05b2488" (UID: "5e6ffe65-3463-40a2-abd5-cf7ee05b2488"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:36:27.785231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.785193 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e6ffe65-3463-40a2-abd5-cf7ee05b2488" (UID: "5e6ffe65-3463-40a2-abd5-cf7ee05b2488"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:36:27.785231 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.785215 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-kube-api-access-65nzm" (OuterVolumeSpecName: "kube-api-access-65nzm") pod "5e6ffe65-3463-40a2-abd5-cf7ee05b2488" (UID: "5e6ffe65-3463-40a2-abd5-cf7ee05b2488"). InnerVolumeSpecName "kube-api-access-65nzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:36:27.785377 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.785249 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e6ffe65-3463-40a2-abd5-cf7ee05b2488" (UID: "5e6ffe65-3463-40a2-abd5-cf7ee05b2488"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:36:27.884576 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.884538 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65nzm\" (UniqueName: \"kubernetes.io/projected/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-kube-api-access-65nzm\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:27.884576 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.884570 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:27.884576 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.884583 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:27.884820 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:27.884595 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ffe65-3463-40a2-abd5-cf7ee05b2488-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:28.028087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:28.028056 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9"] Sep 10 17:36:28.032296 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:28.032271 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2549d0-6d86bb4c88-wgtx9"] Sep 10 17:36:28.040304 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:28.040277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:28.487451 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:28.487419 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6ffe65-3463-40a2-abd5-cf7ee05b2488" path="/var/lib/kubelet/pods/5e6ffe65-3463-40a2-abd5-cf7ee05b2488/volumes" Sep 10 17:36:34.213308 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:34.213273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:34.312072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:34.312032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:34.522811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:34.522723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:34.572660 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:34.572633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:36:34.580219 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:34.580196 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:36:35.117659 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:35.117629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:35.359956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:35.359908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:37.692460 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:37.692422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:50.567572 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.567533 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc"] Sep 10 17:36:50.568142 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.567787 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" podUID="058a05ce-aefd-4594-9f89-cca90a842229" containerName="maas-api" containerID="cri-o://483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe" gracePeriod=30 Sep 10 17:36:50.822707 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.822638 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:36:50.842466 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.842433 2576 generic.go:356] "Generic (PLEG): container finished" podID="058a05ce-aefd-4594-9f89-cca90a842229" containerID="483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe" exitCode=0 Sep 10 17:36:50.842672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.842483 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" Sep 10 17:36:50.842672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.842500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" event={"ID":"058a05ce-aefd-4594-9f89-cca90a842229","Type":"ContainerDied","Data":"483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe"} Sep 10 17:36:50.842672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.842558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc" event={"ID":"058a05ce-aefd-4594-9f89-cca90a842229","Type":"ContainerDied","Data":"d1ed43d12c6e38bdf53906169d32ca10bcfccc90b5499b027072b4284a51f37d"} Sep 10 17:36:50.842672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.842585 2576 scope.go:117] "RemoveContainer" containerID="483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe" Sep 10 17:36:50.854310 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.854286 2576 scope.go:117] "RemoveContainer" containerID="483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe" Sep 10 17:36:50.854707 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:36:50.854680 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe\": container with ID starting with 483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe not found: ID does not exist" containerID="483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe" Sep 10 17:36:50.854810 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.854723 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe"} err="failed to get container status \"483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe\": rpc error: code = NotFound desc = could not find container \"483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe\": container with ID starting with 483e8d54d4d3421c47bd4fadecfcc323c9199af4c6cea413241ae69d6aa569fe not found: ID does not exist" Sep 10 17:36:50.910335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.910303 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbvlz\" (UniqueName: \"kubernetes.io/projected/058a05ce-aefd-4594-9f89-cca90a842229-kube-api-access-fbvlz\") pod \"058a05ce-aefd-4594-9f89-cca90a842229\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " Sep 10 17:36:50.910497 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.910344 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-metrics-certs\") pod \"058a05ce-aefd-4594-9f89-cca90a842229\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " Sep 10 17:36:50.910497 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.910430 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a05ce-aefd-4594-9f89-cca90a842229-openshift-service-ca\") pod \"058a05ce-aefd-4594-9f89-cca90a842229\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " Sep 10 17:36:50.910497 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.910461 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls\") pod \"058a05ce-aefd-4594-9f89-cca90a842229\" (UID: \"058a05ce-aefd-4594-9f89-cca90a842229\") " Sep 10 17:36:50.910874 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.910847 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058a05ce-aefd-4594-9f89-cca90a842229-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "058a05ce-aefd-4594-9f89-cca90a842229" (UID: "058a05ce-aefd-4594-9f89-cca90a842229"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:36:50.912580 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.912552 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "058a05ce-aefd-4594-9f89-cca90a842229" (UID: "058a05ce-aefd-4594-9f89-cca90a842229"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:36:50.912835 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.912812 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "058a05ce-aefd-4594-9f89-cca90a842229" (UID: "058a05ce-aefd-4594-9f89-cca90a842229"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:36:50.912943 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:50.912922 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058a05ce-aefd-4594-9f89-cca90a842229-kube-api-access-fbvlz" (OuterVolumeSpecName: "kube-api-access-fbvlz") pod "058a05ce-aefd-4594-9f89-cca90a842229" (UID: "058a05ce-aefd-4594-9f89-cca90a842229"). InnerVolumeSpecName "kube-api-access-fbvlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:36:51.011553 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.011514 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a05ce-aefd-4594-9f89-cca90a842229-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:51.011553 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.011553 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:51.011823 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.011566 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbvlz\" (UniqueName: \"kubernetes.io/projected/058a05ce-aefd-4594-9f89-cca90a842229-kube-api-access-fbvlz\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:51.011823 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.011578 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058a05ce-aefd-4594-9f89-cca90a842229-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:36:51.173331 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.173302 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc"] Sep 10 17:36:51.179633 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.179603 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65fac0-7d7f9c5d54-55wtc"] Sep 10 17:36:51.299340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.299287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:51.360926 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:51.360866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:52.487304 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:52.487268 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058a05ce-aefd-4594-9f89-cca90a842229" path="/var/lib/kubelet/pods/058a05ce-aefd-4594-9f89-cca90a842229/volumes" Sep 10 17:36:57.476326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:57.476207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:57.576284 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:57.576245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:57.740942 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:57.740839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:36:58.198536 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:36:58.198506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:00.680507 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:00.680473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:02.356374 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:02.356331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:03.475488 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:03.475452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:03.883043 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:03.883004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:04.186937 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:04.186820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:04.559552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:04.559465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:07.486717 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:07.486682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:07.594550 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:07.594507 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:08.090107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:08.090075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:08.816069 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:08.816033 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2"] Sep 10 17:37:08.816522 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:08.816303 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" podUID="3aa57572-1acd-4bd2-a75e-d9613e3526b6" containerName="maas-api" containerID="cri-o://fd8d097c51570323ff73ceccb692801c5abdf71d5f178574a9ec26d8be9bfc0b" gracePeriod=30 Sep 10 17:37:08.843122 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:08.843091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:08.932424 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:08.932391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:08.947288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:08.947254 2576 generic.go:356] "Generic (PLEG): container finished" podID="3aa57572-1acd-4bd2-a75e-d9613e3526b6" containerID="fd8d097c51570323ff73ceccb692801c5abdf71d5f178574a9ec26d8be9bfc0b" exitCode=0 Sep 10 17:37:08.947288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:08.947320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" event={"ID":"3aa57572-1acd-4bd2-a75e-d9613e3526b6","Type":"ContainerDied","Data":"fd8d097c51570323ff73ceccb692801c5abdf71d5f178574a9ec26d8be9bfc0b"} Sep 10 17:37:09.002088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.002046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.074747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.074688 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:37:09.101310 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.101275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.169909 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.169858 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8"] Sep 10 17:37:09.170527 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170504 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa57572-1acd-4bd2-a75e-d9613e3526b6" containerName="maas-api" Sep 10 17:37:09.170527 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170524 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa57572-1acd-4bd2-a75e-d9613e3526b6" containerName="maas-api" Sep 10 17:37:09.170527 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170532 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058a05ce-aefd-4594-9f89-cca90a842229" containerName="maas-api" Sep 10 17:37:09.170785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170538 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="058a05ce-aefd-4594-9f89-cca90a842229" containerName="maas-api" Sep 10 17:37:09.170785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170553 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6ffe65-3463-40a2-abd5-cf7ee05b2488" containerName="maas-api" Sep 10 17:37:09.170785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170558 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6ffe65-3463-40a2-abd5-cf7ee05b2488" containerName="maas-api" Sep 10 17:37:09.170785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170635 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="058a05ce-aefd-4594-9f89-cca90a842229" containerName="maas-api" Sep 10 17:37:09.170785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170645 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6ffe65-3463-40a2-abd5-cf7ee05b2488" containerName="maas-api" Sep 10 17:37:09.170785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.170653 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa57572-1acd-4bd2-a75e-d9613e3526b6" containerName="maas-api" Sep 10 17:37:09.174511 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.174490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.177139 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.177122 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-437dd7-serving-cert\"" Sep 10 17:37:09.184632 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.184604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8"] Sep 10 17:37:09.187761 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.187727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.201181 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.201156 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-metrics-certs\") pod \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " Sep 10 17:37:09.201318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.201199 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57572-1acd-4bd2-a75e-d9613e3526b6-openshift-service-ca\") pod \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " Sep 10 17:37:09.201318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.201280 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l44dc\" (UniqueName: \"kubernetes.io/projected/3aa57572-1acd-4bd2-a75e-d9613e3526b6-kube-api-access-l44dc\") pod \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " Sep 10 17:37:09.201318 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.201315 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-maas-api-tls\") pod \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\" (UID: \"3aa57572-1acd-4bd2-a75e-d9613e3526b6\") " Sep 10 17:37:09.201650 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.201623 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa57572-1acd-4bd2-a75e-d9613e3526b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3aa57572-1acd-4bd2-a75e-d9613e3526b6" (UID: "3aa57572-1acd-4bd2-a75e-d9613e3526b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:37:09.203643 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.203621 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa57572-1acd-4bd2-a75e-d9613e3526b6-kube-api-access-l44dc" (OuterVolumeSpecName: "kube-api-access-l44dc") pod "3aa57572-1acd-4bd2-a75e-d9613e3526b6" (UID: "3aa57572-1acd-4bd2-a75e-d9613e3526b6"). InnerVolumeSpecName "kube-api-access-l44dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:37:09.203920 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.203854 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3aa57572-1acd-4bd2-a75e-d9613e3526b6" (UID: "3aa57572-1acd-4bd2-a75e-d9613e3526b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:37:09.203920 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.203870 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3aa57572-1acd-4bd2-a75e-d9613e3526b6" (UID: "3aa57572-1acd-4bd2-a75e-d9613e3526b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:37:09.280087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.280038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.303473 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-metrics-certs\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.303809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4zt\" (UniqueName: \"kubernetes.io/projected/32ff91ad-ee4c-477d-8b07-e03916936cc4-kube-api-access-pm4zt\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.303809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.303809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ff91ad-ee4c-477d-8b07-e03916936cc4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.304072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303832 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44dc\" (UniqueName: \"kubernetes.io/projected/3aa57572-1acd-4bd2-a75e-d9613e3526b6-kube-api-access-l44dc\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:09.304072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303849 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:09.304072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303861 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa57572-1acd-4bd2-a75e-d9613e3526b6-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:09.304072 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.303868 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57572-1acd-4bd2-a75e-d9613e3526b6-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:09.405432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.405335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ff91ad-ee4c-477d-8b07-e03916936cc4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.405432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.405395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-metrics-certs\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.405432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.405430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4zt\" (UniqueName: \"kubernetes.io/projected/32ff91ad-ee4c-477d-8b07-e03916936cc4-kube-api-access-pm4zt\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.405734 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.405463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.405734 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:37:09.405558 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-serving-cert: secret "maas-api-e2e-ipp-a-437dd7-serving-cert" not found Sep 10 17:37:09.405734 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:37:09.405613 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls podName:32ff91ad-ee4c-477d-8b07-e03916936cc4 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:09.905598068 +0000 UTC m=+1535.886678044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls") pod "maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" (UID: "32ff91ad-ee4c-477d-8b07-e03916936cc4") : secret "maas-api-e2e-ipp-a-437dd7-serving-cert" not found Sep 10 17:37:09.406183 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.406158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ff91ad-ee4c-477d-8b07-e03916936cc4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.407951 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.407934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-metrics-certs\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.415956 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.415935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4zt\" (UniqueName: \"kubernetes.io/projected/32ff91ad-ee4c-477d-8b07-e03916936cc4-kube-api-access-pm4zt\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.467281 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.467252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.717028 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.716945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.856390 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.856358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.910607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.910569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.913003 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.912981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:09.950125 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.950097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:09.954772 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.954751 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" Sep 10 17:37:09.954920 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.954786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2" event={"ID":"3aa57572-1acd-4bd2-a75e-d9613e3526b6","Type":"ContainerDied","Data":"9ffe1840cec349ee3657d0402911c682b7efb35bf40c4974c10b4e750f7736b6"} Sep 10 17:37:09.954920 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:09.954854 2576 scope.go:117] "RemoveContainer" containerID="fd8d097c51570323ff73ceccb692801c5abdf71d5f178574a9ec26d8be9bfc0b" Sep 10 17:37:10.001569 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.001481 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2"] Sep 10 17:37:10.012545 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.012511 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-33b7bb-58c4b559d4-9t2g2"] Sep 10 17:37:10.088622 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.088589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:10.376197 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.376174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8"] Sep 10 17:37:10.379232 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:37:10.379200 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ff91ad_ee4c_477d_8b07_e03916936cc4.slice/crio-e9b99b43924df55f5a4fdf267a11d7177be36aa71df57846257b458be975f895 WatchSource:0}: Error finding container e9b99b43924df55f5a4fdf267a11d7177be36aa71df57846257b458be975f895: Status 404 returned error can't find the container with id e9b99b43924df55f5a4fdf267a11d7177be36aa71df57846257b458be975f895 Sep 10 17:37:10.487517 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.487487 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa57572-1acd-4bd2-a75e-d9613e3526b6" path="/var/lib/kubelet/pods/3aa57572-1acd-4bd2-a75e-d9613e3526b6/volumes" Sep 10 17:37:10.963120 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.963077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" event={"ID":"32ff91ad-ee4c-477d-8b07-e03916936cc4","Type":"ContainerStarted","Data":"d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3"} Sep 10 17:37:10.963502 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.963138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" event={"ID":"32ff91ad-ee4c-477d-8b07-e03916936cc4","Type":"ContainerStarted","Data":"e9b99b43924df55f5a4fdf267a11d7177be36aa71df57846257b458be975f895"} Sep 10 17:37:10.963502 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.963177 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:10.986248 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:10.986147 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" podStartSLOduration=1.6467209120000001 podStartE2EDuration="1.986131764s" totalImagesPullingTime="339.410852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:10.98583199 +0000 UTC m=+1536.966911979" watchObservedRunningTime="2026-09-10 17:37:10.986131764 +0000 UTC m=+1536.967211761" Sep 10 17:37:15.142892 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:15.142858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:15.210258 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:15.210220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:15.411719 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:15.411633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:15.928422 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:15.928382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:16.975363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:16.975328 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:37:26.326659 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.326618 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9"] Sep 10 17:37:26.327115 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.326863 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" podUID="21b44521-ebe9-43ff-be02-017de75aabcf" containerName="maas-api" containerID="cri-o://db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793" gracePeriod=30 Sep 10 17:37:26.351363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.351284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:26.452042 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.452006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:26.551075 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.551039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:26.584324 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.584270 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:37:26.651874 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.651833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:26.671758 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.671730 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-metrics-certs\") pod \"21b44521-ebe9-43ff-be02-017de75aabcf\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " Sep 10 17:37:26.671925 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.671824 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q9vx\" (UniqueName: \"kubernetes.io/projected/21b44521-ebe9-43ff-be02-017de75aabcf-kube-api-access-2q9vx\") pod \"21b44521-ebe9-43ff-be02-017de75aabcf\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " Sep 10 17:37:26.671925 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.671859 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b44521-ebe9-43ff-be02-017de75aabcf-openshift-service-ca\") pod \"21b44521-ebe9-43ff-be02-017de75aabcf\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " Sep 10 17:37:26.672017 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.671995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls\") pod \"21b44521-ebe9-43ff-be02-017de75aabcf\" (UID: \"21b44521-ebe9-43ff-be02-017de75aabcf\") " Sep 10 17:37:26.672274 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.672239 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b44521-ebe9-43ff-be02-017de75aabcf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21b44521-ebe9-43ff-be02-017de75aabcf" (UID: "21b44521-ebe9-43ff-be02-017de75aabcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:37:26.672373 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.672359 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b44521-ebe9-43ff-be02-017de75aabcf-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:26.674161 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.674128 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21b44521-ebe9-43ff-be02-017de75aabcf" (UID: "21b44521-ebe9-43ff-be02-017de75aabcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:37:26.674281 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.674169 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b44521-ebe9-43ff-be02-017de75aabcf-kube-api-access-2q9vx" (OuterVolumeSpecName: "kube-api-access-2q9vx") pod "21b44521-ebe9-43ff-be02-017de75aabcf" (UID: "21b44521-ebe9-43ff-be02-017de75aabcf"). InnerVolumeSpecName "kube-api-access-2q9vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:37:26.674281 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.674229 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "21b44521-ebe9-43ff-be02-017de75aabcf" (UID: "21b44521-ebe9-43ff-be02-017de75aabcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:37:26.773058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.773023 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:26.773058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.773051 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b44521-ebe9-43ff-be02-017de75aabcf-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:26.773058 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.773060 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q9vx\" (UniqueName: \"kubernetes.io/projected/21b44521-ebe9-43ff-be02-017de75aabcf-kube-api-access-2q9vx\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:37:26.952191 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:26.952060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:27.057254 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.057221 2576 generic.go:356] "Generic (PLEG): container finished" podID="21b44521-ebe9-43ff-be02-017de75aabcf" containerID="db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793" exitCode=0 Sep 10 17:37:27.057442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.057265 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" Sep 10 17:37:27.057442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.057305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" event={"ID":"21b44521-ebe9-43ff-be02-017de75aabcf","Type":"ContainerDied","Data":"db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793"} Sep 10 17:37:27.057442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.057360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9" event={"ID":"21b44521-ebe9-43ff-be02-017de75aabcf","Type":"ContainerDied","Data":"771f06682999dde8caa66618529a651575240e86cc0cf1e810476bf753adec1f"} Sep 10 17:37:27.057442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.057391 2576 scope.go:117] "RemoveContainer" containerID="db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793" Sep 10 17:37:27.067010 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.066993 2576 scope.go:117] "RemoveContainer" containerID="db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793" Sep 10 17:37:27.067271 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:37:27.067253 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793\": container with ID starting with db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793 not found: ID does not exist" containerID="db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793" Sep 10 17:37:27.067341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.067282 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793"} err="failed to get container status \"db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793\": rpc error: code = NotFound desc = could not find container \"db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793\": container with ID starting with db3be3da1c6f23d10570fb666e9bc9babd94e7f7d184b27602a159eb6aff1793 not found: ID does not exist" Sep 10 17:37:27.080690 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.080664 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9"] Sep 10 17:37:27.084927 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:27.084907 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-edbcb5-54d8475cc7-d59z9"] Sep 10 17:37:28.487507 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:28.487470 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b44521-ebe9-43ff-be02-017de75aabcf" path="/var/lib/kubelet/pods/21b44521-ebe9-43ff-be02-017de75aabcf/volumes" Sep 10 17:37:30.487127 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:30.487086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:32.434682 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:32.434643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:33.077626 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:33.077587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:33.173242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:33.173204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:33.335867 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:33.335784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:33.417605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:33.417566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:33.775211 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:33.775118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:36.681453 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:36.681399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:36.776502 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:36.776466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:37.795770 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.795738 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn"] Sep 10 17:37:37.796306 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.796286 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b44521-ebe9-43ff-be02-017de75aabcf" containerName="maas-api" Sep 10 17:37:37.796357 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.796309 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b44521-ebe9-43ff-be02-017de75aabcf" containerName="maas-api" Sep 10 17:37:37.796392 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.796378 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b44521-ebe9-43ff-be02-017de75aabcf" containerName="maas-api" Sep 10 17:37:37.799428 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.799408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.802798 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.802765 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-777ae3-serving-cert\"" Sep 10 17:37:37.807729 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.807704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn"] Sep 10 17:37:37.828572 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.828541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:37.879548 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.879513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f791c0-6968-491c-aead-76b5c48b3a74-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.879710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.879558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-metrics-certs\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.879710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.879575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgczr\" (UniqueName: \"kubernetes.io/projected/c8f791c0-6968-491c-aead-76b5c48b3a74-kube-api-access-mgczr\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.879710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.879703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.980468 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.980425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.980677 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.980475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f791c0-6968-491c-aead-76b5c48b3a74-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.980677 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.980513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-metrics-certs\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.980677 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.980537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgczr\" (UniqueName: \"kubernetes.io/projected/c8f791c0-6968-491c-aead-76b5c48b3a74-kube-api-access-mgczr\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.980677 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:37:37.980576 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-serving-cert: secret "maas-api-e2e-ipp-b-777ae3-serving-cert" not found Sep 10 17:37:37.980677 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:37:37.980640 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls podName:c8f791c0-6968-491c-aead-76b5c48b3a74 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:38.480621474 +0000 UTC m=+1564.461701450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls") pod "maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" (UID: "c8f791c0-6968-491c-aead-76b5c48b3a74") : secret "maas-api-e2e-ipp-b-777ae3-serving-cert" not found Sep 10 17:37:37.981278 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.981260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f791c0-6968-491c-aead-76b5c48b3a74-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.982972 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.982953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-metrics-certs\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:37.992116 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:37.992095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgczr\" (UniqueName: \"kubernetes.io/projected/c8f791c0-6968-491c-aead-76b5c48b3a74-kube-api-access-mgczr\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:38.485035 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:38.485004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:38.487637 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:38.487599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls\") pod \"maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:38.712955 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:38.712915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:38.995029 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:38.994994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn"] Sep 10 17:37:38.996149 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:37:38.996121 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f791c0_6968_491c_aead_76b5c48b3a74.slice/crio-2d2baca90b9218aaf956f6e9688e9eb011757fac549232cbfb26915e7b694428 WatchSource:0}: Error finding container 2d2baca90b9218aaf956f6e9688e9eb011757fac549232cbfb26915e7b694428: Status 404 returned error can't find the container with id 2d2baca90b9218aaf956f6e9688e9eb011757fac549232cbfb26915e7b694428 Sep 10 17:37:39.058039 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:39.057959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:39.128481 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:39.128450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" event={"ID":"c8f791c0-6968-491c-aead-76b5c48b3a74","Type":"ContainerStarted","Data":"2d2baca90b9218aaf956f6e9688e9eb011757fac549232cbfb26915e7b694428"} Sep 10 17:37:40.136218 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:40.136184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" event={"ID":"c8f791c0-6968-491c-aead-76b5c48b3a74","Type":"ContainerStarted","Data":"5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674"} Sep 10 17:37:40.136593 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:40.136320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:37:40.154168 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:40.154110 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" podStartSLOduration=2.731221255 podStartE2EDuration="3.154092497s" totalImagesPullingTime="422.871242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:40.153948443 +0000 UTC m=+1566.135028443" watchObservedRunningTime="2026-09-10 17:37:40.154092497 +0000 UTC m=+1566.135172495" Sep 10 17:37:45.786605 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:45.786555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:37:46.148889 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:37:46.148860 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:38:01.646257 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:01.646219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:20.355119 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:20.355086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:20.425287 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:20.425229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:23.968799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:23.968765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:53.070494 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:53.070438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:53.383625 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:53.383537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:53.645128 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:53.645040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:56.202844 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:56.202789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:56.294295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:56.294252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:57.315701 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.315670 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd"] Sep 10 17:38:57.320157 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.320132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.323832 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.323796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-5eebb3-serving-cert\"" Sep 10 17:38:57.330442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.330420 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd"] Sep 10 17:38:57.377341 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.377311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:57.383483 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.383457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74194300-3f5c-4391-8c8b-aa11441e6347-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.383629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.383509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hxs\" (UniqueName: \"kubernetes.io/projected/74194300-3f5c-4391-8c8b-aa11441e6347-kube-api-access-d5hxs\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.383629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.383541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.383629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.383573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.475503 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.475469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:57.485463 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.485433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.485640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.485483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.485640 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.485576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74194300-3f5c-4391-8c8b-aa11441e6347-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.485640 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:38:57.485586 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-serving-cert: secret "maas-api-e2e-ipp-cleanup-5eebb3-serving-cert" not found Sep 10 17:38:57.485803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.485639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hxs\" (UniqueName: \"kubernetes.io/projected/74194300-3f5c-4391-8c8b-aa11441e6347-kube-api-access-d5hxs\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.485803 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:38:57.485664 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls podName:74194300-3f5c-4391-8c8b-aa11441e6347 nodeName:}" failed. No retries permitted until 2026-09-10 17:38:57.985644061 +0000 UTC m=+1643.966724036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" (UID: "74194300-3f5c-4391-8c8b-aa11441e6347") : secret "maas-api-e2e-ipp-cleanup-5eebb3-serving-cert" not found Sep 10 17:38:57.486214 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.486193 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74194300-3f5c-4391-8c8b-aa11441e6347-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.488147 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.488121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.498246 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.498220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hxs\" (UniqueName: \"kubernetes.io/projected/74194300-3f5c-4391-8c8b-aa11441e6347-kube-api-access-d5hxs\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.598618 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.598582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:57.971337 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.971249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:57.990853 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.990823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:57.993432 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:57.993400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:58.066301 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:58.066264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:38:58.235160 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:58.235074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:58.517506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:58.517482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd"] Sep 10 17:38:58.519328 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:38:58.519302 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74194300_3f5c_4391_8c8b_aa11441e6347.slice/crio-a43db317aa4b92496d3b01c43dfd178521da21c3a5e099cf2ecea08150432f48 WatchSource:0}: Error finding container a43db317aa4b92496d3b01c43dfd178521da21c3a5e099cf2ecea08150432f48: Status 404 returned error can't find the container with id a43db317aa4b92496d3b01c43dfd178521da21c3a5e099cf2ecea08150432f48 Sep 10 17:38:58.591344 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:58.591311 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" event={"ID":"74194300-3f5c-4391-8c8b-aa11441e6347","Type":"ContainerStarted","Data":"a43db317aa4b92496d3b01c43dfd178521da21c3a5e099cf2ecea08150432f48"} Sep 10 17:38:59.598536 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:59.598493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" event={"ID":"74194300-3f5c-4391-8c8b-aa11441e6347","Type":"ContainerStarted","Data":"632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73"} Sep 10 17:38:59.599029 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:59.598568 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:38:59.617532 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:38:59.617484 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" podStartSLOduration=2.297874061 podStartE2EDuration="2.617468885s" totalImagesPullingTime="319.594824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:59.616289242 +0000 UTC m=+1645.597369240" watchObservedRunningTime="2026-09-10 17:38:59.617468885 +0000 UTC m=+1645.598548933" Sep 10 17:39:05.610571 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:05.610538 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:39:21.058811 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:21.058769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:27.925493 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:27.925457 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd"] Sep 10 17:39:27.926004 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:27.925737 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" podUID="74194300-3f5c-4391-8c8b-aa11441e6347" containerName="maas-api" containerID="cri-o://632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73" gracePeriod=30 Sep 10 17:39:27.950958 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:27.950924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:28.023153 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.023087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:28.087792 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.087737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:28.193200 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.193134 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:39:28.194730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.194688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:28.298284 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.298250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:28.301199 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.301181 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-metrics-certs\") pod \"74194300-3f5c-4391-8c8b-aa11441e6347\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " Sep 10 17:39:28.301282 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.301268 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5hxs\" (UniqueName: \"kubernetes.io/projected/74194300-3f5c-4391-8c8b-aa11441e6347-kube-api-access-d5hxs\") pod \"74194300-3f5c-4391-8c8b-aa11441e6347\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " Sep 10 17:39:28.301368 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.301339 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74194300-3f5c-4391-8c8b-aa11441e6347-openshift-service-ca\") pod \"74194300-3f5c-4391-8c8b-aa11441e6347\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " Sep 10 17:39:28.301368 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.301363 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls\") pod \"74194300-3f5c-4391-8c8b-aa11441e6347\" (UID: \"74194300-3f5c-4391-8c8b-aa11441e6347\") " Sep 10 17:39:28.301709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.301654 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74194300-3f5c-4391-8c8b-aa11441e6347-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74194300-3f5c-4391-8c8b-aa11441e6347" (UID: "74194300-3f5c-4391-8c8b-aa11441e6347"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:39:28.303479 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.303456 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74194300-3f5c-4391-8c8b-aa11441e6347" (UID: "74194300-3f5c-4391-8c8b-aa11441e6347"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:39:28.303902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.303856 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "74194300-3f5c-4391-8c8b-aa11441e6347" (UID: "74194300-3f5c-4391-8c8b-aa11441e6347"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:39:28.303902 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.303861 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74194300-3f5c-4391-8c8b-aa11441e6347-kube-api-access-d5hxs" (OuterVolumeSpecName: "kube-api-access-d5hxs") pod "74194300-3f5c-4391-8c8b-aa11441e6347" (UID: "74194300-3f5c-4391-8c8b-aa11441e6347"). InnerVolumeSpecName "kube-api-access-d5hxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:39:28.402875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.402833 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74194300-3f5c-4391-8c8b-aa11441e6347-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:28.402875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.402868 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:28.402875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.402903 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74194300-3f5c-4391-8c8b-aa11441e6347-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:28.403146 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.402911 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5hxs\" (UniqueName: \"kubernetes.io/projected/74194300-3f5c-4391-8c8b-aa11441e6347-kube-api-access-d5hxs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:28.596366 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.596333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:28.779253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.779160 2576 generic.go:356] "Generic (PLEG): container finished" podID="74194300-3f5c-4391-8c8b-aa11441e6347" containerID="632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73" exitCode=0 Sep 10 17:39:28.779253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.779202 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" Sep 10 17:39:28.779253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.779240 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" event={"ID":"74194300-3f5c-4391-8c8b-aa11441e6347","Type":"ContainerDied","Data":"632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73"} Sep 10 17:39:28.779600 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.779300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd" event={"ID":"74194300-3f5c-4391-8c8b-aa11441e6347","Type":"ContainerDied","Data":"a43db317aa4b92496d3b01c43dfd178521da21c3a5e099cf2ecea08150432f48"} Sep 10 17:39:28.779600 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.779327 2576 scope.go:117] "RemoveContainer" containerID="632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73" Sep 10 17:39:28.788980 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.788960 2576 scope.go:117] "RemoveContainer" containerID="632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73" Sep 10 17:39:28.789254 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:39:28.789236 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73\": container with ID starting with 632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73 not found: ID does not exist" containerID="632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73" Sep 10 17:39:28.789299 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.789262 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73"} err="failed to get container status \"632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73\": rpc error: code = NotFound desc = could not find container \"632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73\": container with ID starting with 632bb725d5b6947d1dfcb672a5a33a97ef1f2118467ae65594b6fed5943ecc73 not found: ID does not exist" Sep 10 17:39:28.797542 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.797516 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd"] Sep 10 17:39:28.803158 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:28.803136 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5eebb3-68bbc8c6-chzbd"] Sep 10 17:39:30.487760 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:30.487726 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74194300-3f5c-4391-8c8b-aa11441e6347" path="/var/lib/kubelet/pods/74194300-3f5c-4391-8c8b-aa11441e6347/volumes" Sep 10 17:39:35.146938 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:35.146897 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:35.263725 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:35.263678 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:35.486379 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:35.486294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:39.240183 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:39.240146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:54.658388 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.658355 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8"] Sep 10 17:39:54.658799 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.658573 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" podUID="32ff91ad-ee4c-477d-8b07-e03916936cc4" containerName="maas-api" containerID="cri-o://d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3" gracePeriod=30 Sep 10 17:39:54.686990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.686948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:54.790342 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.790301 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:54.911824 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.911744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:54.917452 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.917427 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:39:54.927297 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.927269 2576 generic.go:356] "Generic (PLEG): container finished" podID="32ff91ad-ee4c-477d-8b07-e03916936cc4" containerID="d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3" exitCode=0 Sep 10 17:39:54.927431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.927307 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" Sep 10 17:39:54.927431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.927336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" event={"ID":"32ff91ad-ee4c-477d-8b07-e03916936cc4","Type":"ContainerDied","Data":"d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3"} Sep 10 17:39:54.927431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.927380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8" event={"ID":"32ff91ad-ee4c-477d-8b07-e03916936cc4","Type":"ContainerDied","Data":"e9b99b43924df55f5a4fdf267a11d7177be36aa71df57846257b458be975f895"} Sep 10 17:39:54.927431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.927403 2576 scope.go:117] "RemoveContainer" containerID="d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3" Sep 10 17:39:54.937981 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.937960 2576 scope.go:117] "RemoveContainer" containerID="d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3" Sep 10 17:39:54.938368 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:39:54.938337 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3\": container with ID starting with d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3 not found: ID does not exist" containerID="d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3" Sep 10 17:39:54.938448 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.938405 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3"} err="failed to get container status \"d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3\": rpc error: code = NotFound desc = could not find container \"d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3\": container with ID starting with d5dd01aea6ce992d03870faa44bb9cf7733a1079a27f2e340388044bd267b4a3 not found: ID does not exist" Sep 10 17:39:54.961959 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.961870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls\") pod \"32ff91ad-ee4c-477d-8b07-e03916936cc4\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " Sep 10 17:39:54.962387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.962036 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ff91ad-ee4c-477d-8b07-e03916936cc4-openshift-service-ca\") pod \"32ff91ad-ee4c-477d-8b07-e03916936cc4\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " Sep 10 17:39:54.962387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.962075 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm4zt\" (UniqueName: \"kubernetes.io/projected/32ff91ad-ee4c-477d-8b07-e03916936cc4-kube-api-access-pm4zt\") pod \"32ff91ad-ee4c-477d-8b07-e03916936cc4\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " Sep 10 17:39:54.962387 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.962123 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-metrics-certs\") pod \"32ff91ad-ee4c-477d-8b07-e03916936cc4\" (UID: \"32ff91ad-ee4c-477d-8b07-e03916936cc4\") " Sep 10 17:39:54.963687 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.963005 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ff91ad-ee4c-477d-8b07-e03916936cc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32ff91ad-ee4c-477d-8b07-e03916936cc4" (UID: "32ff91ad-ee4c-477d-8b07-e03916936cc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:39:54.967778 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.967713 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ff91ad-ee4c-477d-8b07-e03916936cc4-kube-api-access-pm4zt" (OuterVolumeSpecName: "kube-api-access-pm4zt") pod "32ff91ad-ee4c-477d-8b07-e03916936cc4" (UID: "32ff91ad-ee4c-477d-8b07-e03916936cc4"). InnerVolumeSpecName "kube-api-access-pm4zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:39:54.967940 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.967907 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32ff91ad-ee4c-477d-8b07-e03916936cc4" (UID: "32ff91ad-ee4c-477d-8b07-e03916936cc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:39:54.968332 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:54.968306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32ff91ad-ee4c-477d-8b07-e03916936cc4" (UID: "32ff91ad-ee4c-477d-8b07-e03916936cc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:39:55.014301 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.014262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:55.063953 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.063911 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ff91ad-ee4c-477d-8b07-e03916936cc4-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:55.063953 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.063947 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm4zt\" (UniqueName: \"kubernetes.io/projected/32ff91ad-ee4c-477d-8b07-e03916936cc4-kube-api-access-pm4zt\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:55.063953 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.063962 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:55.064198 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.063975 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ff91ad-ee4c-477d-8b07-e03916936cc4-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:39:55.111506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.111468 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:55.255773 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.255739 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8"] Sep 10 17:39:55.258321 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.258299 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-437dd7-75f54886d8-x2kt8"] Sep 10 17:39:55.453029 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:55.452992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:39:56.487340 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:39:56.487306 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ff91ad-ee4c-477d-8b07-e03916936cc4" path="/var/lib/kubelet/pods/32ff91ad-ee4c-477d-8b07-e03916936cc4/volumes" Sep 10 17:40:00.498336 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:00.498255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:01.185565 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:01.185529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:29.817553 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:29.817500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:37.226263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:37.226221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:37.533807 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:37.533723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:43.174846 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:43.174808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:43.240893 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:43.240844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:43.454803 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:43.454721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:43.906127 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:43.906089 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:54.312335 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.312303 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn"] Sep 10 17:40:54.312738 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.312521 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" podUID="c8f791c0-6968-491c-aead-76b5c48b3a74" containerName="maas-api" containerID="cri-o://5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674" gracePeriod=30 Sep 10 17:40:54.347866 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.347831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:54.416150 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.416110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:54.474213 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.474175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:54.552002 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.551969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:54.557203 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.557176 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:40:54.621073 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.621042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:54.661061 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.661024 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls\") pod \"c8f791c0-6968-491c-aead-76b5c48b3a74\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " Sep 10 17:40:54.661256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.661074 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f791c0-6968-491c-aead-76b5c48b3a74-openshift-service-ca\") pod \"c8f791c0-6968-491c-aead-76b5c48b3a74\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " Sep 10 17:40:54.661256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.661152 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgczr\" (UniqueName: \"kubernetes.io/projected/c8f791c0-6968-491c-aead-76b5c48b3a74-kube-api-access-mgczr\") pod \"c8f791c0-6968-491c-aead-76b5c48b3a74\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " Sep 10 17:40:54.661256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.661193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-metrics-certs\") pod \"c8f791c0-6968-491c-aead-76b5c48b3a74\" (UID: \"c8f791c0-6968-491c-aead-76b5c48b3a74\") " Sep 10 17:40:54.661528 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.661499 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f791c0-6968-491c-aead-76b5c48b3a74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8f791c0-6968-491c-aead-76b5c48b3a74" (UID: "c8f791c0-6968-491c-aead-76b5c48b3a74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:40:54.663256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.663231 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8f791c0-6968-491c-aead-76b5c48b3a74" (UID: "c8f791c0-6968-491c-aead-76b5c48b3a74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:40:54.663364 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.663276 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8f791c0-6968-491c-aead-76b5c48b3a74" (UID: "c8f791c0-6968-491c-aead-76b5c48b3a74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:40:54.663364 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.663327 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f791c0-6968-491c-aead-76b5c48b3a74-kube-api-access-mgczr" (OuterVolumeSpecName: "kube-api-access-mgczr") pod "c8f791c0-6968-491c-aead-76b5c48b3a74" (UID: "c8f791c0-6968-491c-aead-76b5c48b3a74"). InnerVolumeSpecName "kube-api-access-mgczr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:54.762756 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.762727 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:40:54.762756 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.762751 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f791c0-6968-491c-aead-76b5c48b3a74-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:40:54.762756 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.762761 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgczr\" (UniqueName: \"kubernetes.io/projected/c8f791c0-6968-491c-aead-76b5c48b3a74-kube-api-access-mgczr\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:40:54.763005 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.762769 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f791c0-6968-491c-aead-76b5c48b3a74-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:40:54.923806 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:54.923727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:40:55.286875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.286795 2576 generic.go:356] "Generic (PLEG): container finished" podID="c8f791c0-6968-491c-aead-76b5c48b3a74" containerID="5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674" exitCode=0 Sep 10 17:40:55.286875 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.286840 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" Sep 10 17:40:55.287087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.286903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" event={"ID":"c8f791c0-6968-491c-aead-76b5c48b3a74","Type":"ContainerDied","Data":"5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674"} Sep 10 17:40:55.287087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.286953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn" event={"ID":"c8f791c0-6968-491c-aead-76b5c48b3a74","Type":"ContainerDied","Data":"2d2baca90b9218aaf956f6e9688e9eb011757fac549232cbfb26915e7b694428"} Sep 10 17:40:55.287087 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.286975 2576 scope.go:117] "RemoveContainer" containerID="5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674" Sep 10 17:40:55.296967 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.296951 2576 scope.go:117] "RemoveContainer" containerID="5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674" Sep 10 17:40:55.297203 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:40:55.297184 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674\": container with ID starting with 5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674 not found: ID does not exist" containerID="5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674" Sep 10 17:40:55.297256 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.297211 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674"} err="failed to get container status \"5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674\": rpc error: code = NotFound desc = could not find container \"5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674\": container with ID starting with 5f59cccde06709ea622c90044b905dedff75a6d3d3d8fb3ddb0a7e2face7a674 not found: ID does not exist" Sep 10 17:40:55.309749 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.309723 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn"] Sep 10 17:40:55.313457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:55.313436 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-777ae3-6f8cc5c89d-rrhvn"] Sep 10 17:40:56.487431 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:40:56.487396 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f791c0-6968-491c-aead-76b5c48b3a74" path="/var/lib/kubelet/pods/c8f791c0-6968-491c-aead-76b5c48b3a74/volumes" Sep 10 17:41:00.987696 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:00.987665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:01.057472 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:01.057434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:01.264779 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:01.264667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:01.695861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:01.695825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:12.159921 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.159869 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d"] Sep 10 17:41:12.160361 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.160167 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" podUID="7420c73c-7447-4337-811d-54acd98bd47f" containerName="maas-api" containerID="cri-o://2e75660dfb945a865f1466e7200d410399eb0d32d05a3b95f9f35e26088cb316" gracePeriod=30 Sep 10 17:41:12.197088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.197046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:12.297460 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.297413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:12.387500 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.387467 2576 generic.go:356] "Generic (PLEG): container finished" podID="7420c73c-7447-4337-811d-54acd98bd47f" containerID="2e75660dfb945a865f1466e7200d410399eb0d32d05a3b95f9f35e26088cb316" exitCode=0 Sep 10 17:41:12.387672 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.387541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" event={"ID":"7420c73c-7447-4337-811d-54acd98bd47f","Type":"ContainerDied","Data":"2e75660dfb945a865f1466e7200d410399eb0d32d05a3b95f9f35e26088cb316"} Sep 10 17:41:12.407112 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.407079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:12.410998 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.410941 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:41:12.426331 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.426302 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7420c73c-7447-4337-811d-54acd98bd47f-openshift-service-ca\") pod \"7420c73c-7447-4337-811d-54acd98bd47f\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " Sep 10 17:41:12.426523 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.426357 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-metrics-certs\") pod \"7420c73c-7447-4337-811d-54acd98bd47f\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " Sep 10 17:41:12.426523 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.426414 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls\") pod \"7420c73c-7447-4337-811d-54acd98bd47f\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " Sep 10 17:41:12.426523 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.426432 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njf9n\" (UniqueName: \"kubernetes.io/projected/7420c73c-7447-4337-811d-54acd98bd47f-kube-api-access-njf9n\") pod \"7420c73c-7447-4337-811d-54acd98bd47f\" (UID: \"7420c73c-7447-4337-811d-54acd98bd47f\") " Sep 10 17:41:12.426755 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.426733 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7420c73c-7447-4337-811d-54acd98bd47f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7420c73c-7447-4337-811d-54acd98bd47f" (UID: "7420c73c-7447-4337-811d-54acd98bd47f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:41:12.426864 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.426848 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7420c73c-7447-4337-811d-54acd98bd47f-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:12.428644 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.428622 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7420c73c-7447-4337-811d-54acd98bd47f" (UID: "7420c73c-7447-4337-811d-54acd98bd47f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:41:12.428763 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.428640 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7420c73c-7447-4337-811d-54acd98bd47f" (UID: "7420c73c-7447-4337-811d-54acd98bd47f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:41:12.428763 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.428656 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7420c73c-7447-4337-811d-54acd98bd47f-kube-api-access-njf9n" (OuterVolumeSpecName: "kube-api-access-njf9n") pod "7420c73c-7447-4337-811d-54acd98bd47f" (UID: "7420c73c-7447-4337-811d-54acd98bd47f"). InnerVolumeSpecName "kube-api-access-njf9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:41:12.485187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.485151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:12.528450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.528409 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:12.528450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.528445 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7420c73c-7447-4337-811d-54acd98bd47f-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:12.528450 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.528458 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njf9n\" (UniqueName: \"kubernetes.io/projected/7420c73c-7447-4337-811d-54acd98bd47f-kube-api-access-njf9n\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:12.831466 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:12.831419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:13.394705 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:13.394676 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" Sep 10 17:41:13.395196 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:13.394722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d" event={"ID":"7420c73c-7447-4337-811d-54acd98bd47f","Type":"ContainerDied","Data":"c638f97e93a4e90b0184866769d2fea0abb7b12aa341107a5020bea8aaaf293d"} Sep 10 17:41:13.395196 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:13.394785 2576 scope.go:117] "RemoveContainer" containerID="2e75660dfb945a865f1466e7200d410399eb0d32d05a3b95f9f35e26088cb316" Sep 10 17:41:13.411785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:13.411759 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d"] Sep 10 17:41:13.415922 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:13.415900 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-be583c-7fb8cd4766-4672d"] Sep 10 17:41:14.487261 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:14.487216 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7420c73c-7447-4337-811d-54acd98bd47f" path="/var/lib/kubelet/pods/7420c73c-7447-4337-811d-54acd98bd47f/volumes" Sep 10 17:41:18.796442 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:18.796405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:18.870527 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:18.870472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:19.055491 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:19.055395 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7"] Sep 10 17:41:19.055694 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:19.055667 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" containerID="cri-o://addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4" gracePeriod=30 Sep 10 17:41:19.085292 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:19.085265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:19.693933 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:19.693891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:24.313722 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.313696 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:41:24.438671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438590 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-data\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.438671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438625 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-token\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.438671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438643 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istiod-ca-cert\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.438671 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438659 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-credential-socket\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.439063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438732 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-envoy\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.439063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438756 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-socket\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.439063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438795 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-certs\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.439063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hngm9\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-kube-api-access-hngm9\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.439063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:24.439063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.438960 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-podinfo\") pod \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\" (UID: \"8e68049f-cc72-4cd6-bb18-50bdf02c834c\") " Sep 10 17:41:24.439363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439103 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:24.439363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439131 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:24.439363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439151 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:41:24.439363 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439272 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:24.439579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439385 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.439579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439406 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-data\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.439579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439420 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istiod-ca-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.439579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439433 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-credential-socket\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.439579 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.439448 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-workload-socket\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.440983 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.440952 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:41:24.441102 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.441069 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:41:24.441197 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.441177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:24.441284 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.441265 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-kube-api-access-hngm9" (OuterVolumeSpecName: "kube-api-access-hngm9") pod "8e68049f-cc72-4cd6-bb18-50bdf02c834c" (UID: "8e68049f-cc72-4cd6-bb18-50bdf02c834c"). InnerVolumeSpecName "kube-api-access-hngm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:41:24.457818 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.457794 2576 generic.go:356] "Generic (PLEG): container finished" podID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerID="addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4" exitCode=0 Sep 10 17:41:24.457926 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.457832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" event={"ID":"8e68049f-cc72-4cd6-bb18-50bdf02c834c","Type":"ContainerDied","Data":"addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4"} Sep 10 17:41:24.457926 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.457841 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" Sep 10 17:41:24.457926 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.457902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7" event={"ID":"8e68049f-cc72-4cd6-bb18-50bdf02c834c","Type":"ContainerDied","Data":"61185d357d2c056898d7f9f0111d067aa4fac6b665e465c14bbcddf0ad3b8763"} Sep 10 17:41:24.457926 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.457926 2576 scope.go:117] "RemoveContainer" containerID="addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4" Sep 10 17:41:24.467013 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.466996 2576 scope.go:117] "RemoveContainer" containerID="0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da" Sep 10 17:41:24.475113 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.475098 2576 scope.go:117] "RemoveContainer" containerID="addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4" Sep 10 17:41:24.475366 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:41:24.475348 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4\": container with ID starting with addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4 not found: ID does not exist" containerID="addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4" Sep 10 17:41:24.475438 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.475380 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4"} err="failed to get container status \"addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4\": rpc error: code = NotFound desc = could not find container \"addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4\": container with ID starting with addb8e472142096b56968c8bf2f376384b6a339165606bd650d74d07d0a9fdf4 not found: ID does not exist" Sep 10 17:41:24.475438 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.475403 2576 scope.go:117] "RemoveContainer" containerID="0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da" Sep 10 17:41:24.475651 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:41:24.475632 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da\": container with ID starting with 0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da not found: ID does not exist" containerID="0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da" Sep 10 17:41:24.475691 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.475657 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da"} err="failed to get container status \"0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da\": rpc error: code = NotFound desc = could not find container \"0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da\": container with ID starting with 0664c9d73b40fa0a5054b2ad9e9dcd5e8eb31f3051ecd2ab7992b37cad9216da not found: ID does not exist" Sep 10 17:41:24.487228 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.487210 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7"] Sep 10 17:41:24.487316 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.487238 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-be583c-openshift-default-876c8858d-76gc7"] Sep 10 17:41:24.540457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.540428 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-podinfo\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.540457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.540451 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-token\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.540457 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.540461 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e68049f-cc72-4cd6-bb18-50bdf02c834c-istio-envoy\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:24.540698 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:24.540469 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hngm9\" (UniqueName: \"kubernetes.io/projected/8e68049f-cc72-4cd6-bb18-50bdf02c834c-kube-api-access-hngm9\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:26.487394 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:26.487360 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" path="/var/lib/kubelet/pods/8e68049f-cc72-4cd6-bb18-50bdf02c834c/volumes" Sep 10 17:41:29.990453 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:29.990374 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w"] Sep 10 17:41:29.990776 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:29.990612 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" podUID="a56f58e1-d3da-4b2b-a691-810c3f546eb9" containerName="maas-api" containerID="cri-o://df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21" gracePeriod=30 Sep 10 17:41:30.013537 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.013507 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:30.116705 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.116670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:30.218222 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.218145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:30.242347 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.242288 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:41:30.304109 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.304069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:30.394688 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.394661 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls\") pod \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " Sep 10 17:41:30.394860 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.394697 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-metrics-certs\") pod \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " Sep 10 17:41:30.394860 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.394724 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56f58e1-d3da-4b2b-a691-810c3f546eb9-openshift-service-ca\") pod \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " Sep 10 17:41:30.394860 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.394768 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rtpg\" (UniqueName: \"kubernetes.io/projected/a56f58e1-d3da-4b2b-a691-810c3f546eb9-kube-api-access-2rtpg\") pod \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\" (UID: \"a56f58e1-d3da-4b2b-a691-810c3f546eb9\") " Sep 10 17:41:30.395103 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.395069 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56f58e1-d3da-4b2b-a691-810c3f546eb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a56f58e1-d3da-4b2b-a691-810c3f546eb9" (UID: "a56f58e1-d3da-4b2b-a691-810c3f546eb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:41:30.395226 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.395207 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56f58e1-d3da-4b2b-a691-810c3f546eb9-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:30.396974 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.396947 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a56f58e1-d3da-4b2b-a691-810c3f546eb9" (UID: "a56f58e1-d3da-4b2b-a691-810c3f546eb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:41:30.397074 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.397062 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a56f58e1-d3da-4b2b-a691-810c3f546eb9" (UID: "a56f58e1-d3da-4b2b-a691-810c3f546eb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:41:30.397185 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.397157 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56f58e1-d3da-4b2b-a691-810c3f546eb9-kube-api-access-2rtpg" (OuterVolumeSpecName: "kube-api-access-2rtpg") pod "a56f58e1-d3da-4b2b-a691-810c3f546eb9" (UID: "a56f58e1-d3da-4b2b-a691-810c3f546eb9"). InnerVolumeSpecName "kube-api-access-2rtpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:41:30.494861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.494784 2576 generic.go:356] "Generic (PLEG): container finished" podID="a56f58e1-d3da-4b2b-a691-810c3f546eb9" containerID="df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21" exitCode=0 Sep 10 17:41:30.494861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.494833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" event={"ID":"a56f58e1-d3da-4b2b-a691-810c3f546eb9","Type":"ContainerDied","Data":"df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21"} Sep 10 17:41:30.494861 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.494837 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" Sep 10 17:41:30.495126 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.494902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w" event={"ID":"a56f58e1-d3da-4b2b-a691-810c3f546eb9","Type":"ContainerDied","Data":"22488bea169b6876626c744c97fd498db13a44e631c5b6f34c5e6620a10e8dda"} Sep 10 17:41:30.495126 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.494933 2576 scope.go:117] "RemoveContainer" containerID="df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21" Sep 10 17:41:30.495760 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.495731 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rtpg\" (UniqueName: \"kubernetes.io/projected/a56f58e1-d3da-4b2b-a691-810c3f546eb9-kube-api-access-2rtpg\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:30.495894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.495764 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:30.495894 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.495779 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a56f58e1-d3da-4b2b-a691-810c3f546eb9-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:30.504606 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.504587 2576 scope.go:117] "RemoveContainer" containerID="df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21" Sep 10 17:41:30.504889 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:41:30.504855 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21\": container with ID starting with df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21 not found: ID does not exist" containerID="df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21" Sep 10 17:41:30.504996 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.504900 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21"} err="failed to get container status \"df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21\": rpc error: code = NotFound desc = could not find container \"df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21\": container with ID starting with df8d1858566d4a8f7fd29c2cf889f98561b721d8e748881ac5255f47e68acd21 not found: ID does not exist" Sep 10 17:41:30.515075 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.515055 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w"] Sep 10 17:41:30.518982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.518961 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6806f1-78556bfcc6-wb42w"] Sep 10 17:41:30.665597 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:30.665563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:32.486574 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:32.486540 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56f58e1-d3da-4b2b-a691-810c3f546eb9" path="/var/lib/kubelet/pods/a56f58e1-d3da-4b2b-a691-810c3f546eb9/volumes" Sep 10 17:41:34.610816 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:34.610792 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:41:34.619308 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:34.619285 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:41:37.187770 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:37.187730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:37.268895 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:37.268856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:37.474384 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:37.474303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:39.921395 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:39.921359 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6768fddc-l8k2k"] Sep 10 17:41:39.921846 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:39.921603 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b6768fddc-l8k2k" podUID="d78a65f8-e841-4928-a7d5-3ac276472c77" containerName="manager" containerID="cri-o://78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe" gracePeriod=30 Sep 10 17:41:40.123587 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.123550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:41:40.174076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.174023 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:41:40.281554 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.281527 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx458\" (UniqueName: \"kubernetes.io/projected/d78a65f8-e841-4928-a7d5-3ac276472c77-kube-api-access-mx458\") pod \"d78a65f8-e841-4928-a7d5-3ac276472c77\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " Sep 10 17:41:40.281723 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.281601 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-metrics-certs\") pod \"d78a65f8-e841-4928-a7d5-3ac276472c77\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " Sep 10 17:41:40.281723 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.281654 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-webhook-cert\") pod \"d78a65f8-e841-4928-a7d5-3ac276472c77\" (UID: \"d78a65f8-e841-4928-a7d5-3ac276472c77\") " Sep 10 17:41:40.283551 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.283523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78a65f8-e841-4928-a7d5-3ac276472c77-kube-api-access-mx458" (OuterVolumeSpecName: "kube-api-access-mx458") pod "d78a65f8-e841-4928-a7d5-3ac276472c77" (UID: "d78a65f8-e841-4928-a7d5-3ac276472c77"). InnerVolumeSpecName "kube-api-access-mx458". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:41:40.283551 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.283537 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d78a65f8-e841-4928-a7d5-3ac276472c77" (UID: "d78a65f8-e841-4928-a7d5-3ac276472c77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:41:40.283685 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.283553 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d78a65f8-e841-4928-a7d5-3ac276472c77" (UID: "d78a65f8-e841-4928-a7d5-3ac276472c77"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:41:40.382999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.382966 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-webhook-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:40.382999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.382994 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx458\" (UniqueName: \"kubernetes.io/projected/d78a65f8-e841-4928-a7d5-3ac276472c77-kube-api-access-mx458\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:40.382999 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.383004 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d78a65f8-e841-4928-a7d5-3ac276472c77-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:41:40.559709 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.559675 2576 generic.go:356] "Generic (PLEG): container finished" podID="d78a65f8-e841-4928-a7d5-3ac276472c77" containerID="78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe" exitCode=0 Sep 10 17:41:40.559872 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.559726 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-l8k2k" Sep 10 17:41:40.559872 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.559763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-l8k2k" event={"ID":"d78a65f8-e841-4928-a7d5-3ac276472c77","Type":"ContainerDied","Data":"78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe"} Sep 10 17:41:40.559872 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.559818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-l8k2k" event={"ID":"d78a65f8-e841-4928-a7d5-3ac276472c77","Type":"ContainerDied","Data":"18e3b7ae85aaa8bdb7a70e3eafb8ded04a17cdb9a5ee59ae27b4b24e06c56839"} Sep 10 17:41:40.559872 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.559842 2576 scope.go:117] "RemoveContainer" containerID="78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe" Sep 10 17:41:40.568915 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.568898 2576 scope.go:117] "RemoveContainer" containerID="78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe" Sep 10 17:41:40.569136 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:41:40.569118 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe\": container with ID starting with 78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe not found: ID does not exist" containerID="78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe" Sep 10 17:41:40.569187 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.569144 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe"} err="failed to get container status \"78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe\": rpc error: code = NotFound desc = could not find container \"78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe\": container with ID starting with 78d16b77781b8043a5e532fd1d60827a9150261e6126800cdbbd5816b53ae2fe not found: ID does not exist" Sep 10 17:41:40.578841 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.578818 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6768fddc-l8k2k"] Sep 10 17:41:40.583673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:40.583647 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b6768fddc-l8k2k"] Sep 10 17:41:42.487183 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:41:42.487149 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78a65f8-e841-4928-a7d5-3ac276472c77" path="/var/lib/kubelet/pods/d78a65f8-e841-4928-a7d5-3ac276472c77/volumes" Sep 10 17:42:41.377214 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377176 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6768fddc-wtw27"] Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377535 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ff91ad-ee4c-477d-8b07-e03916936cc4" containerName="maas-api" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377546 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ff91ad-ee4c-477d-8b07-e03916936cc4" containerName="maas-api" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377554 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f791c0-6968-491c-aead-76b5c48b3a74" containerName="maas-api" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377559 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f791c0-6968-491c-aead-76b5c48b3a74" containerName="maas-api" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377575 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377582 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377593 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7420c73c-7447-4337-811d-54acd98bd47f" containerName="maas-api" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377598 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7420c73c-7447-4337-811d-54acd98bd47f" containerName="maas-api" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377604 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377609 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" Sep 10 17:42:41.377611 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377616 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56f58e1-d3da-4b2b-a691-810c3f546eb9" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377621 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56f58e1-d3da-4b2b-a691-810c3f546eb9" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377629 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78a65f8-e841-4928-a7d5-3ac276472c77" containerName="manager" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377634 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78a65f8-e841-4928-a7d5-3ac276472c77" containerName="manager" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377639 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74194300-3f5c-4391-8c8b-aa11441e6347" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377645 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="74194300-3f5c-4391-8c8b-aa11441e6347" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377708 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7420c73c-7447-4337-811d-54acd98bd47f" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377715 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f791c0-6968-491c-aead-76b5c48b3a74" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377722 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377727 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ff91ad-ee4c-477d-8b07-e03916936cc4" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377734 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e68049f-cc72-4cd6-bb18-50bdf02c834c" containerName="istio-proxy" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377739 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56f58e1-d3da-4b2b-a691-810c3f546eb9" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377744 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="74194300-3f5c-4391-8c8b-aa11441e6347" containerName="maas-api" Sep 10 17:42:41.377990 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.377748 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78a65f8-e841-4928-a7d5-3ac276472c77" containerName="manager" Sep 10 17:42:41.380813 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.380795 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.383642 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.383620 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 17:42:41.385040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.385020 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 17:42:41.385040 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.385028 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 17:42:41.385273 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.385077 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8v7fk\"" Sep 10 17:42:41.392038 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.392013 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6768fddc-wtw27"] Sep 10 17:42:41.422784 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.422760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-metrics-certs\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.422917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.422806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-webhook-cert\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.422917 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.422835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr92q\" (UniqueName: \"kubernetes.io/projected/89579bc0-7080-46c3-ac59-86374f8aa4b3-kube-api-access-xr92q\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.524311 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.524268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-metrics-certs\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.524479 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.524329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-webhook-cert\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.524479 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.524355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr92q\" (UniqueName: \"kubernetes.io/projected/89579bc0-7080-46c3-ac59-86374f8aa4b3-kube-api-access-xr92q\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.526845 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.526822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-webhook-cert\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.526959 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.526845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-metrics-certs\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.533144 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.533124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr92q\" (UniqueName: \"kubernetes.io/projected/89579bc0-7080-46c3-ac59-86374f8aa4b3-kube-api-access-xr92q\") pod \"maas-controller-b6768fddc-wtw27\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.692154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.692077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:41.965504 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.965476 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6768fddc-wtw27"] Sep 10 17:42:41.966271 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:42:41.966245 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89579bc0_7080_46c3_ac59_86374f8aa4b3.slice/crio-9c580a5e357834d32ee339920dcce6accdab9ef6385976598e6a574ab9f957be WatchSource:0}: Error finding container 9c580a5e357834d32ee339920dcce6accdab9ef6385976598e6a574ab9f957be: Status 404 returned error can't find the container with id 9c580a5e357834d32ee339920dcce6accdab9ef6385976598e6a574ab9f957be Sep 10 17:42:41.967607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:41.967587 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:42:42.891190 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:42.891151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-wtw27" event={"ID":"89579bc0-7080-46c3-ac59-86374f8aa4b3","Type":"ContainerStarted","Data":"63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a"} Sep 10 17:42:42.891544 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:42.891212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-wtw27" event={"ID":"89579bc0-7080-46c3-ac59-86374f8aa4b3","Type":"ContainerStarted","Data":"9c580a5e357834d32ee339920dcce6accdab9ef6385976598e6a574ab9f957be"} Sep 10 17:42:42.891544 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:42.891281 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:42:42.911736 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:42.911680 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6768fddc-wtw27" podStartSLOduration=1.3745224839999999 podStartE2EDuration="1.911663722s" totalImagesPullingTime="537.141238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:42.910617221 +0000 UTC m=+1868.891697219" watchObservedRunningTime="2026-09-10 17:42:42.911663722 +0000 UTC m=+1868.892743717" Sep 10 17:42:51.934470 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:51.934429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:42:53.899802 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:42:53.899768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:43:01.688309 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:01.688215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:43:03.128351 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.128314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:43:03.236053 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.236015 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6768fddc-wtw27"] Sep 10 17:43:03.236321 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.236278 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b6768fddc-wtw27" podUID="89579bc0-7080-46c3-ac59-86374f8aa4b3" containerName="manager" containerID="cri-o://63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a" gracePeriod=30 Sep 10 17:43:03.508326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.508292 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:43:03.623552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.623522 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-webhook-cert\") pod \"89579bc0-7080-46c3-ac59-86374f8aa4b3\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " Sep 10 17:43:03.623751 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.623564 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-metrics-certs\") pod \"89579bc0-7080-46c3-ac59-86374f8aa4b3\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " Sep 10 17:43:03.623751 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.623668 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr92q\" (UniqueName: \"kubernetes.io/projected/89579bc0-7080-46c3-ac59-86374f8aa4b3-kube-api-access-xr92q\") pod \"89579bc0-7080-46c3-ac59-86374f8aa4b3\" (UID: \"89579bc0-7080-46c3-ac59-86374f8aa4b3\") " Sep 10 17:43:03.625834 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.625775 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "89579bc0-7080-46c3-ac59-86374f8aa4b3" (UID: "89579bc0-7080-46c3-ac59-86374f8aa4b3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:43:03.626326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.626301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89579bc0-7080-46c3-ac59-86374f8aa4b3" (UID: "89579bc0-7080-46c3-ac59-86374f8aa4b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:43:03.626433 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.626368 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89579bc0-7080-46c3-ac59-86374f8aa4b3-kube-api-access-xr92q" (OuterVolumeSpecName: "kube-api-access-xr92q") pod "89579bc0-7080-46c3-ac59-86374f8aa4b3" (UID: "89579bc0-7080-46c3-ac59-86374f8aa4b3"). InnerVolumeSpecName "kube-api-access-xr92q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:43:03.724976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.724874 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-webhook-cert\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:43:03.724976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.724917 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89579bc0-7080-46c3-ac59-86374f8aa4b3-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:43:03.724976 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:03.724927 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr92q\" (UniqueName: \"kubernetes.io/projected/89579bc0-7080-46c3-ac59-86374f8aa4b3-kube-api-access-xr92q\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:43:04.005552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.005468 2576 generic.go:356] "Generic (PLEG): container finished" podID="89579bc0-7080-46c3-ac59-86374f8aa4b3" containerID="63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a" exitCode=0 Sep 10 17:43:04.005552 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.005516 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-wtw27" Sep 10 17:43:04.005730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.005552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-wtw27" event={"ID":"89579bc0-7080-46c3-ac59-86374f8aa4b3","Type":"ContainerDied","Data":"63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a"} Sep 10 17:43:04.005730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.005607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-wtw27" event={"ID":"89579bc0-7080-46c3-ac59-86374f8aa4b3","Type":"ContainerDied","Data":"9c580a5e357834d32ee339920dcce6accdab9ef6385976598e6a574ab9f957be"} Sep 10 17:43:04.005730 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.005629 2576 scope.go:117] "RemoveContainer" containerID="63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a" Sep 10 17:43:04.015255 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.015243 2576 scope.go:117] "RemoveContainer" containerID="63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a" Sep 10 17:43:04.015520 ip-10-0-137-217 kubenswrapper[2576]: E0910 17:43:04.015503 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a\": container with ID starting with 63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a not found: ID does not exist" containerID="63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a" Sep 10 17:43:04.015567 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.015526 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a"} err="failed to get container status \"63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a\": rpc error: code = NotFound desc = could not find container \"63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a\": container with ID starting with 63bb2fb31d03583c9e4812594d6f10102b8da77d848354fbaf5e26c8a011035a not found: ID does not exist" Sep 10 17:43:04.030076 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.030050 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6768fddc-wtw27"] Sep 10 17:43:04.033590 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.033565 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b6768fddc-wtw27"] Sep 10 17:43:04.488376 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.488346 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89579bc0-7080-46c3-ac59-86374f8aa4b3" path="/var/lib/kubelet/pods/89579bc0-7080-46c3-ac59-86374f8aa4b3/volumes" Sep 10 17:43:04.694290 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.694256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:43:04.801681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.801596 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6768fddc-26lst"] Sep 10 17:43:04.802106 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.802089 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89579bc0-7080-46c3-ac59-86374f8aa4b3" containerName="manager" Sep 10 17:43:04.802106 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.802107 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="89579bc0-7080-46c3-ac59-86374f8aa4b3" containerName="manager" Sep 10 17:43:04.802263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.802203 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="89579bc0-7080-46c3-ac59-86374f8aa4b3" containerName="manager" Sep 10 17:43:04.806681 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.806659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:04.810969 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.810951 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 17:43:04.811088 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.810953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 17:43:04.811216 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.811203 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 17:43:04.811279 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.811259 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8v7fk\"" Sep 10 17:43:04.817738 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.817719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6768fddc-26lst"] Sep 10 17:43:04.936775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.936738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpg6\" (UniqueName: \"kubernetes.io/projected/c2958709-95f8-41a4-baba-f2c9df365e0e-kube-api-access-4tpg6\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:04.936964 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.936801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2958709-95f8-41a4-baba-f2c9df365e0e-webhook-cert\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:04.936964 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:04.936905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2958709-95f8-41a4-baba-f2c9df365e0e-metrics-certs\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.037982 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.037950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2958709-95f8-41a4-baba-f2c9df365e0e-webhook-cert\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.038125 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.038011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2958709-95f8-41a4-baba-f2c9df365e0e-metrics-certs\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.038125 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.038093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpg6\" (UniqueName: \"kubernetes.io/projected/c2958709-95f8-41a4-baba-f2c9df365e0e-kube-api-access-4tpg6\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.040430 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.040399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2958709-95f8-41a4-baba-f2c9df365e0e-metrics-certs\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.040538 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.040444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2958709-95f8-41a4-baba-f2c9df365e0e-webhook-cert\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.047393 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.047368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpg6\" (UniqueName: \"kubernetes.io/projected/c2958709-95f8-41a4-baba-f2c9df365e0e-kube-api-access-4tpg6\") pod \"maas-controller-b6768fddc-26lst\" (UID: \"c2958709-95f8-41a4-baba-f2c9df365e0e\") " pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.119167 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.119144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:05.395923 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:05.394521 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6768fddc-26lst"] Sep 10 17:43:06.018968 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:06.018933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-26lst" event={"ID":"c2958709-95f8-41a4-baba-f2c9df365e0e","Type":"ContainerStarted","Data":"1b802ac6cf2778087a65ce085d2630a5c4dbfbee1e2a75bffbdda447cf4892f3"} Sep 10 17:43:06.019424 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:06.018984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6768fddc-26lst" event={"ID":"c2958709-95f8-41a4-baba-f2c9df365e0e","Type":"ContainerStarted","Data":"cc385407a449c120f9d881e92df10e06931d319e030524d17a63b8da6a91c506"} Sep 10 17:43:06.019424 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:06.019067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:06.044063 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:06.044014 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6768fddc-26lst" podStartSLOduration=1.6450421610000001 podStartE2EDuration="2.043999203s" totalImagesPullingTime="398.957042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:06.040725717 +0000 UTC m=+1892.021805737" watchObservedRunningTime="2026-09-10 17:43:06.043999203 +0000 UTC m=+1892.025079201" Sep 10 17:43:17.028068 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:17.028030 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6768fddc-26lst" Sep 10 17:43:54.162095 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:43:54.162057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:45:05.812836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:45:05.812801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:46:05.640193 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:05.640096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:46:05.740673 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:05.740633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:46:11.584526 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:11.584495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:46:25.204959 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:25.204923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:46:34.648061 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:34.648032 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:46:34.659171 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:34.659146 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:46:52.599739 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:52.599697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:46:54.588809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:46:54.588763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:47:10.543629 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.543587 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt"] Sep 10 17:47:10.547297 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.547271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.573000 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.572970 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt"] Sep 10 17:47:10.615339 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.615299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee45cab-be1e-4736-a2a0-1c129b999614-maas-api-tls\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.615546 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.615358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qql8j\" (UniqueName: \"kubernetes.io/projected/7ee45cab-be1e-4736-a2a0-1c129b999614-kube-api-access-qql8j\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.615546 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.615448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee45cab-be1e-4736-a2a0-1c129b999614-openshift-service-ca\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.615546 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.615495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee45cab-be1e-4736-a2a0-1c129b999614-metrics-certs\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.716676 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.716637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee45cab-be1e-4736-a2a0-1c129b999614-maas-api-tls\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.716862 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.716727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qql8j\" (UniqueName: \"kubernetes.io/projected/7ee45cab-be1e-4736-a2a0-1c129b999614-kube-api-access-qql8j\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.716862 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.716761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee45cab-be1e-4736-a2a0-1c129b999614-openshift-service-ca\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.716862 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.716793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee45cab-be1e-4736-a2a0-1c129b999614-metrics-certs\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.717506 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.717485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee45cab-be1e-4736-a2a0-1c129b999614-openshift-service-ca\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.719258 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.719231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee45cab-be1e-4736-a2a0-1c129b999614-metrics-certs\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.719371 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.719278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee45cab-be1e-4736-a2a0-1c129b999614-maas-api-tls\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.728952 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.728926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qql8j\" (UniqueName: \"kubernetes.io/projected/7ee45cab-be1e-4736-a2a0-1c129b999614-kube-api-access-qql8j\") pod \"maas-api-777cfbf6dc-k4hzt\" (UID: \"7ee45cab-be1e-4736-a2a0-1c129b999614\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:10.858630 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:10.858596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:11.161551 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:11.161524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt"] Sep 10 17:47:11.163741 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:47:11.163714 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee45cab_be1e_4736_a2a0_1c129b999614.slice/crio-f528a1daea73ffc3fbc338c3963c3d449a7fbe008801e6deb58158c1bbcda9c1 WatchSource:0}: Error finding container f528a1daea73ffc3fbc338c3963c3d449a7fbe008801e6deb58158c1bbcda9c1: Status 404 returned error can't find the container with id f528a1daea73ffc3fbc338c3963c3d449a7fbe008801e6deb58158c1bbcda9c1 Sep 10 17:47:11.379121 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:11.379067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" event={"ID":"7ee45cab-be1e-4736-a2a0-1c129b999614","Type":"ContainerStarted","Data":"f528a1daea73ffc3fbc338c3963c3d449a7fbe008801e6deb58158c1bbcda9c1"} Sep 10 17:47:12.386238 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:12.386197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" event={"ID":"7ee45cab-be1e-4736-a2a0-1c129b999614","Type":"ContainerStarted","Data":"638473896e3bed234478663ca3b6b8d28e77bc63a6fee5b9e911f1ed630386bf"} Sep 10 17:47:12.386662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:12.386282 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:12.408253 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:12.408208 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" podStartSLOduration=2.015611504 podStartE2EDuration="2.40819454s" totalImagesPullingTime="392.583036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:12.407526215 +0000 UTC m=+2138.388606214" watchObservedRunningTime="2026-09-10 17:47:12.40819454 +0000 UTC m=+2138.389274537" Sep 10 17:47:18.405414 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:18.405377 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-k4hzt" Sep 10 17:47:28.376050 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:28.375951 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76"] Sep 10 17:47:28.376578 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:28.376177 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" podUID="2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" containerName="maas-api" containerID="cri-o://38c0ece50b383c8c6d97143c0f546d7a24a00013cfc2c28a4c0e5eaf10bd7236" gracePeriod=30 Sep 10 17:47:29.493263 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.493212 2576 generic.go:356] "Generic (PLEG): container finished" podID="2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" containerID="38c0ece50b383c8c6d97143c0f546d7a24a00013cfc2c28a4c0e5eaf10bd7236" exitCode=0 Sep 10 17:47:29.493565 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.493294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" event={"ID":"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a","Type":"ContainerDied","Data":"38c0ece50b383c8c6d97143c0f546d7a24a00013cfc2c28a4c0e5eaf10bd7236"} Sep 10 17:47:29.628736 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.628713 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:47:29.702052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.701962 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-openshift-service-ca\") pod \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " Sep 10 17:47:29.702052 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.702026 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-maas-api-tls\") pod \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " Sep 10 17:47:29.702280 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.702056 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-metrics-certs\") pod \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " Sep 10 17:47:29.702280 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.702111 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txttf\" (UniqueName: \"kubernetes.io/projected/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-kube-api-access-txttf\") pod \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\" (UID: \"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a\") " Sep 10 17:47:29.702408 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.702385 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" (UID: "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:47:29.702526 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.702508 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-openshift-service-ca\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:47:29.704330 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.704298 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-kube-api-access-txttf" (OuterVolumeSpecName: "kube-api-access-txttf") pod "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" (UID: "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a"). InnerVolumeSpecName "kube-api-access-txttf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:47:29.704458 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.704337 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" (UID: "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:47:29.704458 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.704348 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" (UID: "2e6177e1-985f-4bfc-b224-e97ce0fc1e3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:47:29.803836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.803803 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-maas-api-tls\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:47:29.803836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.803831 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-metrics-certs\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:47:29.803836 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:29.803840 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txttf\" (UniqueName: \"kubernetes.io/projected/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a-kube-api-access-txttf\") on node \"ip-10-0-137-217.ec2.internal\" DevicePath \"\"" Sep 10 17:47:30.502045 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:30.502017 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" Sep 10 17:47:30.502045 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:30.502028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76" event={"ID":"2e6177e1-985f-4bfc-b224-e97ce0fc1e3a","Type":"ContainerDied","Data":"3a4bbbe532fddd07288bc87343c1b5e8d94611e649bf3c290c7e4665a47fe836"} Sep 10 17:47:30.502476 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:30.502103 2576 scope.go:117] "RemoveContainer" containerID="38c0ece50b383c8c6d97143c0f546d7a24a00013cfc2c28a4c0e5eaf10bd7236" Sep 10 17:47:30.522217 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:30.522192 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76"] Sep 10 17:47:30.524644 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:30.524621 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-pgq76"] Sep 10 17:47:32.493652 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:32.493613 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" path="/var/lib/kubelet/pods/2e6177e1-985f-4bfc-b224-e97ce0fc1e3a/volumes" Sep 10 17:47:46.720343 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:46.720310 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lnsmw_98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17/manager/0.log" Sep 10 17:47:46.988192 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:46.988098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b6768fddc-26lst_c2958709-95f8-41a4-baba-f2c9df365e0e/manager/0.log" Sep 10 17:47:47.113606 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:47.113575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tn44v_f1a83d70-1e1f-4b39-b07a-402432beb414/server/0.log" Sep 10 17:47:47.239945 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:47.239841 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wgc26_efbf1d1e-8071-4771-85a4-a283330d3af0/manager/0.log" Sep 10 17:47:47.620810 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:47.620776 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-m2jwr_78a67348-ba67-4210-a24f-e756e5eb8f40/copy-manifests/0.log" Sep 10 17:47:47.629297 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:47.629262 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-m2jwr_78a67348-ba67-4210-a24f-e756e5eb8f40/manager/0.log" Sep 10 17:47:47.976710 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:47.976631 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-777cfbf6dc-k4hzt_7ee45cab-be1e-4736-a2a0-1c129b999614/maas-api/0.log" Sep 10 17:47:48.419113 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:48.419076 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vzqqn_31896f7c-9e3e-46fc-973f-f5025b3afe88/postgres/0.log" Sep 10 17:47:49.533822 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:49.533791 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nlzbf_d6e20548-80e2-43c7-8f64-66167f80e60f/kuadrant-console-plugin/0.log" Sep 10 17:47:50.229977 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:50.229932 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz_6292fd1a-ada8-458b-8486-e24743c1ccb2/istio-proxy/0.log" Sep 10 17:47:50.687114 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:50.687081 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gmhd_4a35ba9e-05d0-4a6a-90dc-03e621ea9e04/istio-proxy/1.log" Sep 10 17:47:51.044025 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:51.043916 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d5b75dfd-l6m8l_99a001fe-c41e-457b-b8ff-64fa4da2af08/router/0.log" Sep 10 17:47:51.381986 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:51.381955 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vrrhm_6bfefb18-2b34-43ef-9708-e75e150d1ff9/main/0.log" Sep 10 17:47:51.719984 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:51.719901 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-d2rnp_55bbd2be-f242-4686-9a55-e177df48f59c/main/0.log" Sep 10 17:47:51.955349 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:51.955319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-p8hxz_4633c4e7-0c65-42db-b36d-fc7b0dfee9f1/main/0.log" Sep 10 17:47:57.167326 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.167289 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mlqxs/must-gather-ss987"] Sep 10 17:47:57.167698 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.167678 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" containerName="maas-api" Sep 10 17:47:57.167698 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.167688 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" containerName="maas-api" Sep 10 17:47:57.167772 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.167757 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6177e1-985f-4bfc-b224-e97ce0fc1e3a" containerName="maas-api" Sep 10 17:47:57.171127 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.171107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.173911 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.173871 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mlqxs\"/\"default-dockercfg-trhqt\"" Sep 10 17:47:57.174122 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.174105 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlqxs\"/\"openshift-service-ca.crt\"" Sep 10 17:47:57.175060 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.175046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlqxs\"/\"kube-root-ca.crt\"" Sep 10 17:47:57.181019 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.181000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlqxs/must-gather-ss987"] Sep 10 17:47:57.191351 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.191328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:47:57.269418 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.269384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/117c174a-c572-4adc-92b8-78dbc8b309f1-must-gather-output\") pod \"must-gather-ss987\" (UID: \"117c174a-c572-4adc-92b8-78dbc8b309f1\") " pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.269418 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.269420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2tf\" (UniqueName: \"kubernetes.io/projected/117c174a-c572-4adc-92b8-78dbc8b309f1-kube-api-access-2k2tf\") pod \"must-gather-ss987\" (UID: \"117c174a-c572-4adc-92b8-78dbc8b309f1\") " pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.370419 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.370384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/117c174a-c572-4adc-92b8-78dbc8b309f1-must-gather-output\") pod \"must-gather-ss987\" (UID: \"117c174a-c572-4adc-92b8-78dbc8b309f1\") " pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.370419 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.370418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2tf\" (UniqueName: \"kubernetes.io/projected/117c174a-c572-4adc-92b8-78dbc8b309f1-kube-api-access-2k2tf\") pod \"must-gather-ss987\" (UID: \"117c174a-c572-4adc-92b8-78dbc8b309f1\") " pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.370775 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.370752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/117c174a-c572-4adc-92b8-78dbc8b309f1-must-gather-output\") pod \"must-gather-ss987\" (UID: \"117c174a-c572-4adc-92b8-78dbc8b309f1\") " pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.378829 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.378804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2tf\" (UniqueName: \"kubernetes.io/projected/117c174a-c572-4adc-92b8-78dbc8b309f1-kube-api-access-2k2tf\") pod \"must-gather-ss987\" (UID: \"117c174a-c572-4adc-92b8-78dbc8b309f1\") " pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.481061 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.480983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlqxs/must-gather-ss987" Sep 10 17:47:57.761619 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.761591 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlqxs/must-gather-ss987"] Sep 10 17:47:57.763109 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:47:57.763072 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod117c174a_c572_4adc_92b8_78dbc8b309f1.slice/crio-11f29abd1231f877b6a23a711d2602f33a2e5b4332e363610f68524b02f360af WatchSource:0}: Error finding container 11f29abd1231f877b6a23a711d2602f33a2e5b4332e363610f68524b02f360af: Status 404 returned error can't find the container with id 11f29abd1231f877b6a23a711d2602f33a2e5b4332e363610f68524b02f360af Sep 10 17:47:57.765107 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:57.765092 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:47:58.365201 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:58.365168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" limit=1024 Sep 10 17:47:58.656728 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:58.656691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlqxs/must-gather-ss987" event={"ID":"117c174a-c572-4adc-92b8-78dbc8b309f1","Type":"ContainerStarted","Data":"11f29abd1231f877b6a23a711d2602f33a2e5b4332e363610f68524b02f360af"} Sep 10 17:47:59.665933 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:59.665868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlqxs/must-gather-ss987" event={"ID":"117c174a-c572-4adc-92b8-78dbc8b309f1","Type":"ContainerStarted","Data":"d179c7ba1723baac1e4a9c599539344bf07c3e0de9d8bfb7743f5790ce9faf6f"} Sep 10 17:47:59.666297 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:59.665949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlqxs/must-gather-ss987" event={"ID":"117c174a-c572-4adc-92b8-78dbc8b309f1","Type":"ContainerStarted","Data":"fa58b447bd14eb2cce0fcbb2ae727b08c65de1495a874c17bc870e18ab09fc6d"} Sep 10 17:47:59.688664 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:47:59.688594 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mlqxs/must-gather-ss987" podStartSLOduration=1.853018621 podStartE2EDuration="2.68856908s" totalImagesPullingTime="835.550459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:59.687726562 +0000 UTC m=+2185.668806561" watchObservedRunningTime="2026-09-10 17:47:59.68856908 +0000 UTC m=+2185.669649093" Sep 10 17:48:00.365143 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:00.365107 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8r6bk_0f7c913b-fedc-4ca8-9d95-fa73663da5d4/global-pull-secret-syncer/0.log" Sep 10 17:48:00.424732 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:00.424693 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b4rjd_af136ee8-5433-4faa-868a-83038e568060/konnectivity-agent/0.log" Sep 10 17:48:00.540086 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:00.540040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-217.ec2.internal_56c1d6772edd98b5770557256f012f4e/haproxy/0.log" Sep 10 17:48:05.308031 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:05.308005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nlzbf_d6e20548-80e2-43c7-8f64-66167f80e60f/kuadrant-console-plugin/0.log" Sep 10 17:48:07.141242 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:07.141214 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-6zv7s_170c6e7a-0aa9-41f8-a475-904a458b67a0/cluster-monitoring-operator/0.log" Sep 10 17:48:07.263903 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:07.263844 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67bcd4c8f7-qn52k_99e3d4da-cacb-4d4d-82ce-2634e4f90cd3/metrics-server/0.log" Sep 10 17:48:07.340200 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:07.340165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dtf5_311a0a72-08d7-422d-81c0-aea6b47ea149/node-exporter/0.log" Sep 10 17:48:07.366294 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:07.366262 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dtf5_311a0a72-08d7-422d-81c0-aea6b47ea149/kube-rbac-proxy/0.log" Sep 10 17:48:07.393018 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:07.392929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dtf5_311a0a72-08d7-422d-81c0-aea6b47ea149/init-textfile/0.log" Sep 10 17:48:07.983154 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:07.983023 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xccsp_ed4e6175-bb3b-4d8b-819a-216728ac1c69/prometheus-operator/0.log" Sep 10 17:48:08.008839 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:08.008799 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xccsp_ed4e6175-bb3b-4d8b-819a-216728ac1c69/kube-rbac-proxy/0.log" Sep 10 17:48:10.356771 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.356735 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/2.log" Sep 10 17:48:10.365568 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.365538 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tv8w9_71a87714-cd43-4cd4-b7a6-d23ad894d1df/console-operator/3.log" Sep 10 17:48:10.489931 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.489900 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf"] Sep 10 17:48:10.495755 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.495730 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.497319 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.497295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf"] Sep 10 17:48:10.626123 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.626029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-proc\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.626288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.626173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-lib-modules\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.626288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.626208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvqn2\" (UniqueName: \"kubernetes.io/projected/0c8e4776-927a-4402-bc71-35ecb999959a-kube-api-access-qvqn2\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.626288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.626260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-sys\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.626288 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.626284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-podres\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.726785 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.726749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-lib-modules\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.726993 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.726791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvqn2\" (UniqueName: \"kubernetes.io/projected/0c8e4776-927a-4402-bc71-35ecb999959a-kube-api-access-qvqn2\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.726993 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.726835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-sys\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.726993 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.726858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-podres\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.726993 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.726953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-lib-modules\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.726993 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.726962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-sys\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.726993 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.726969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-proc\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.727346 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.727061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-podres\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.727346 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.727101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c8e4776-927a-4402-bc71-35ecb999959a-proc\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.737813 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.737779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvqn2\" (UniqueName: \"kubernetes.io/projected/0c8e4776-927a-4402-bc71-35ecb999959a-kube-api-access-qvqn2\") pod \"perf-node-gather-daemonset-zrtbf\" (UID: \"0c8e4776-927a-4402-bc71-35ecb999959a\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.821753 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.821681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:10.967498 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:10.967468 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56bf68d87d-zdpfh_ac274796-d611-46d9-8781-1841abfb12d8/console/0.log" Sep 10 17:48:11.005516 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:11.005486 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-6cvjc_01237739-3e8f-4ded-af03-8f4f92ce821d/download-server/0.log" Sep 10 17:48:11.124782 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:11.124368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf"] Sep 10 17:48:11.132149 ip-10-0-137-217 kubenswrapper[2576]: W0910 17:48:11.130839 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c8e4776_927a_4402_bc71_35ecb999959a.slice/crio-97a31198e8a3b879f0f4b368dca5063a7b4acd7401cbfbc1b27ddac2882701d8 WatchSource:0}: Error finding container 97a31198e8a3b879f0f4b368dca5063a7b4acd7401cbfbc1b27ddac2882701d8: Status 404 returned error can't find the container with id 97a31198e8a3b879f0f4b368dca5063a7b4acd7401cbfbc1b27ddac2882701d8 Sep 10 17:48:11.754593 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:11.752787 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-l2vqj_5fa8506b-482d-456a-bab2-a7c26b3d8e64/volume-data-source-validator/0.log" Sep 10 17:48:11.774963 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:11.774932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" event={"ID":"0c8e4776-927a-4402-bc71-35ecb999959a","Type":"ContainerStarted","Data":"71f966599e52da69072c78a22c0be938804c560fdf73b3ce45bb11def3f1891b"} Sep 10 17:48:11.775194 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:11.774987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" event={"ID":"0c8e4776-927a-4402-bc71-35ecb999959a","Type":"ContainerStarted","Data":"97a31198e8a3b879f0f4b368dca5063a7b4acd7401cbfbc1b27ddac2882701d8"} Sep 10 17:48:11.775194 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:11.775076 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:11.795899 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:11.792656 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" podStartSLOduration=1.792637739 podStartE2EDuration="1.792637739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:11.792291122 +0000 UTC m=+2197.773371122" watchObservedRunningTime="2026-09-10 17:48:11.792637739 +0000 UTC m=+2197.773717739" Sep 10 17:48:12.961217 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:12.961184 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbd5r_a9cc5e36-0e94-4097-b240-bd5660695b52/dns/0.log" Sep 10 17:48:12.988777 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:12.988747 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbd5r_a9cc5e36-0e94-4097-b240-bd5660695b52/kube-rbac-proxy/0.log" Sep 10 17:48:13.016081 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:13.016051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d9247_40439c75-55c7-42f5-9d84-6b380daf2e05/dns-node-resolver/0.log" Sep 10 17:48:13.680433 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:13.680404 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d2rkj_f6feefb4-04cc-497b-870f-609aab9359f9/node-ca/0.log" Sep 10 17:48:14.733508 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:14.733478 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz_6292fd1a-ada8-458b-8486-e24743c1ccb2/istio-proxy/0.log" Sep 10 17:48:15.053747 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:15.053673 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gmhd_4a35ba9e-05d0-4a6a-90dc-03e621ea9e04/istio-proxy/0.log" Sep 10 17:48:15.070391 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:15.070363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gmhd_4a35ba9e-05d0-4a6a-90dc-03e621ea9e04/istio-proxy/1.log" Sep 10 17:48:15.183071 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:15.183043 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d5b75dfd-l6m8l_99a001fe-c41e-457b-b8ff-64fa4da2af08/router/0.log" Sep 10 17:48:16.104205 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:16.104172 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cndb6_7428d8c1-26fb-4a73-b4c2-ff5cee1533cc/serve-healthcheck-canary/0.log" Sep 10 17:48:16.641454 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:16.641406 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-wpxtc_77c8c142-262a-4f8e-aac2-fa61beaee390/insights-operator/0.log" Sep 10 17:48:16.644553 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:16.644524 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-wpxtc_77c8c142-262a-4f8e-aac2-fa61beaee390/insights-operator/1.log" Sep 10 17:48:16.851839 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:16.851807 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lp2hc_19b86683-66b6-4717-bae8-95e9bed10496/kube-rbac-proxy/0.log" Sep 10 17:48:16.877130 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:16.877091 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lp2hc_19b86683-66b6-4717-bae8-95e9bed10496/exporter/0.log" Sep 10 17:48:16.909607 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:16.909521 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lp2hc_19b86683-66b6-4717-bae8-95e9bed10496/extractor/0.log" Sep 10 17:48:17.795226 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:17.795197 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-zrtbf" Sep 10 17:48:19.289631 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:19.289591 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lnsmw_98cbdd4c-bcfa-4ef7-85a4-9e2c40f58a17/manager/0.log" Sep 10 17:48:19.556170 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:19.556092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b6768fddc-26lst_c2958709-95f8-41a4-baba-f2c9df365e0e/manager/0.log" Sep 10 17:48:19.588153 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:19.588127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tn44v_f1a83d70-1e1f-4b39-b07a-402432beb414/server/0.log" Sep 10 17:48:19.737234 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:19.737202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wgc26_efbf1d1e-8071-4771-85a4-a283330d3af0/manager/0.log" Sep 10 17:48:19.958101 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:19.958068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-m2jwr_78a67348-ba67-4210-a24f-e756e5eb8f40/manager/0.log" Sep 10 17:48:19.982324 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:19.982297 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-m2jwr_78a67348-ba67-4210-a24f-e756e5eb8f40/copy-manifests/0.log" Sep 10 17:48:21.385996 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:21.385959 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz_6292fd1a-ada8-458b-8486-e24743c1ccb2/istio-proxy/0.log" Sep 10 17:48:21.480809 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:21.480777 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9fc458d94-ds2ph_c44f1e05-226d-40c9-8626-484e90514cbb/manager/0.log" Sep 10 17:48:21.674929 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:21.674809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gmhd_4a35ba9e-05d0-4a6a-90dc-03e621ea9e04/istio-proxy/0.log" Sep 10 17:48:21.684700 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:21.684666 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gmhd_4a35ba9e-05d0-4a6a-90dc-03e621ea9e04/istio-proxy/1.log" Sep 10 17:48:21.765062 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:21.765030 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d5b75dfd-l6m8l_99a001fe-c41e-457b-b8ff-64fa4da2af08/router/0.log" Sep 10 17:48:23.312295 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:23.312255 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-b8jmz_6292fd1a-ada8-458b-8486-e24743c1ccb2/istio-proxy/0.log" Sep 10 17:48:23.610039 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:23.610007 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gmhd_4a35ba9e-05d0-4a6a-90dc-03e621ea9e04/istio-proxy/0.log" Sep 10 17:48:23.627409 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:23.627384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gmhd_4a35ba9e-05d0-4a6a-90dc-03e621ea9e04/istio-proxy/1.log" Sep 10 17:48:23.705189 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:23.705128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d5b75dfd-l6m8l_99a001fe-c41e-457b-b8ff-64fa4da2af08/router/0.log" Sep 10 17:48:27.448662 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:27.448628 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fxk8j_1a1aa361-e7eb-48f9-82e0-6bb7206e0beb/kube-storage-version-migrator-operator/1.log" Sep 10 17:48:27.450581 ip-10-0-137-217 kubenswrapper[2576]: I0910 17:48:27.450536 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fxk8j_1a1aa361-e7eb-48f9-82e0-6bb7206e0beb/kube-storage-version-migrator-operator/0.log"