Sep 15 20:34:47.237141 ip-10-0-139-201 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 15 20:34:47.237154 ip-10-0-139-201 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 15 20:34:47.237163 ip-10-0-139-201 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 15 20:34:47.237491 ip-10-0-139-201 systemd[1]: Failed to start Kubernetes Kubelet. Sep 15 20:34:57.479992 ip-10-0-139-201 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 15 20:34:57.480013 ip-10-0-139-201 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b6f72e6680ff4591a4b69b4fc5b129e3 -- Sep 15 20:36:50.771702 ip-10-0-139-201 systemd[1]: Starting Kubernetes Kubelet... Sep 15 20:36:51.096844 ip-10-0-139-201 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 20:36:51.096844 ip-10-0-139-201 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 20:36:51.096844 ip-10-0-139-201 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 20:36:51.096844 ip-10-0-139-201 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 20:36:51.096844 ip-10-0-139-201 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 20:36:51.098596 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.098518 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 20:36:51.102913 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102898 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:51.102913 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102913 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:51.102913 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102917 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102920 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102923 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102928 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102931 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102933 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102936 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102939 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102942 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102946 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102950 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102954 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102957 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102959 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102962 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102965 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102968 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102971 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102974 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:51.103010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102977 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102980 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102982 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102985 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102987 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102990 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102993 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102995 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.102997 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103000 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103002 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103005 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103007 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103010 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103013 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103016 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103019 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103022 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103025 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103027 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:51.103477 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103030 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103032 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103035 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103037 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103040 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103042 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103045 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103047 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103050 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103052 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103055 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103057 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103060 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103062 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103067 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103069 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103072 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103075 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103077 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103080 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:51.103964 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103083 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103085 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103088 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103091 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103093 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103096 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103100 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103103 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103106 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103110 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103113 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103116 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103118 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103121 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103124 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103126 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103129 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103131 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103134 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:51.104466 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103136 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103140 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103143 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103146 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103149 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103152 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103155 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103159 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103162 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103165 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103167 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103170 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103173 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103176 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103178 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103181 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103183 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103186 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.103188 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104554 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:51.104953 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104561 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104564 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104567 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104570 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104574 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104576 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104579 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104582 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104584 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104587 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104590 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104592 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104594 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104599 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104603 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104606 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104609 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104612 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104614 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:51.105551 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104618 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104621 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104624 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104627 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104630 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104633 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104635 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104638 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104640 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104642 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104652 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104655 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104658 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104661 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104663 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104666 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104668 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104671 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104674 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:51.106050 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104677 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104679 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104682 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104685 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104688 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104691 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104694 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104696 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104699 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104701 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104704 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104707 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104709 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104712 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104715 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104717 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104721 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104725 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104729 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104731 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:51.106538 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104734 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104737 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104739 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104742 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104745 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104747 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104751 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104753 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104756 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104758 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104761 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104764 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104766 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104769 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104772 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104774 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104777 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104779 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104782 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104784 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:51.107031 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104787 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104789 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104792 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104794 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104797 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104802 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104804 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104807 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104809 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104814 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104817 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104820 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104823 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104828 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104831 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104833 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104836 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104838 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104841 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.104843 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:51.107533 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105204 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105208 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105211 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105213 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105217 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105220 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105222 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105225 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105228 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105231 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105233 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105236 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105238 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105241 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105244 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105247 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105250 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105253 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105256 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105258 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:51.108018 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105260 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105263 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105266 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105270 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105273 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105278 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105281 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105284 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105287 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105290 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105294 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105296 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105299 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105301 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105304 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105306 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105308 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105311 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105315 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:51.108514 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105317 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105319 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105322 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105324 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105327 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105329 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105331 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105334 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105337 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105339 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105342 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105344 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105347 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105349 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105352 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105355 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105358 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105360 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105364 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105366 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:51.108979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105370 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105374 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105376 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105379 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105381 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105384 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105387 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105389 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105392 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105395 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105398 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105401 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105415 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105418 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105429 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105432 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105434 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105437 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105439 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105442 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:51.109476 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105444 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105447 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105450 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105452 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105455 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105458 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105460 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105463 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105465 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105468 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105471 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105475 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105477 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105480 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105482 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105485 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105488 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105490 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105493 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.105495 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:51.110006 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105566 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105598 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105604 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105608 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105613 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105616 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105621 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105625 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105628 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105631 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105635 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105638 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105642 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105645 2580 flags.go:64] FLAG: --cgroup-root="" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105656 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105659 2580 flags.go:64] FLAG: --client-ca-file="" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105662 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105665 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105670 2580 flags.go:64] FLAG: --cluster-domain="" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105673 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105676 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105679 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105682 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105688 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 20:36:51.110522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105691 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105693 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105697 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105700 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105702 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105705 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105708 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105711 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105715 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105718 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105721 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105723 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105726 2580 flags.go:64] FLAG: --enable-server="true" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105729 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105733 2580 flags.go:64] FLAG: --event-burst="100" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105736 2580 flags.go:64] FLAG: --event-qps="50" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105739 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105742 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105745 2580 flags.go:64] FLAG: --eviction-hard="" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105748 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105751 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105755 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105758 2580 flags.go:64] FLAG: --eviction-soft="" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105761 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105764 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 20:36:51.111107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105766 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105769 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105772 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105775 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105778 2580 flags.go:64] FLAG: --feature-gates="" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105781 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105785 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105788 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105791 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105794 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105797 2580 flags.go:64] FLAG: --help="false" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105800 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105803 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105806 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105809 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105812 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105816 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105818 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105821 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105824 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105827 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105830 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105833 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105836 2580 flags.go:64] FLAG: --kube-reserved="" Sep 15 20:36:51.111740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105839 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105841 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105845 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105847 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105851 2580 flags.go:64] FLAG: --lock-file="" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105855 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105858 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105860 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105866 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105868 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105871 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105874 2580 flags.go:64] FLAG: --logging-format="text" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105877 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105880 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105887 2580 flags.go:64] FLAG: --manifest-url="" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105890 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105894 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105897 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105901 2580 flags.go:64] FLAG: --max-pods="110" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105904 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105907 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105910 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105913 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105915 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105918 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 20:36:51.112725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105921 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105928 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105931 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105934 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105937 2580 flags.go:64] FLAG: --pod-cidr="" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105940 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105945 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105948 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105951 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105954 2580 flags.go:64] FLAG: --port="10250" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105957 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105960 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b4f9fd2eec66ddd7" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105964 2580 flags.go:64] FLAG: --qos-reserved="" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105967 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105970 2580 flags.go:64] FLAG: --register-node="true" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105973 2580 flags.go:64] FLAG: --register-with-taints="" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105977 2580 flags.go:64] FLAG: --registry-burst="10" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105979 2580 flags.go:64] FLAG: --registry-qps="5" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105982 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105985 2580 flags.go:64] FLAG: --reserved-memory="" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105988 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105993 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105996 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.105999 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106002 2580 flags.go:64] FLAG: --runonce="false" Sep 15 20:36:51.113824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106005 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106008 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106011 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106014 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106016 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106019 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106022 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106025 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106028 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106031 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106034 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106037 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106040 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106044 2580 flags.go:64] FLAG: --system-cgroups="" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106046 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106052 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106055 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106057 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106062 2580 flags.go:64] FLAG: --tls-min-version="" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106066 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106069 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106071 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106074 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106077 2580 flags.go:64] FLAG: --v="2" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106081 2580 flags.go:64] FLAG: --version="false" Sep 15 20:36:51.114597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106085 2580 flags.go:64] FLAG: --vmodule="" Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106089 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106092 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106217 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106222 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106226 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106229 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106232 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106235 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106238 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106241 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106244 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106247 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106249 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106252 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106255 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106258 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106261 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106264 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106266 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:51.115561 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106269 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106272 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106274 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106276 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106279 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106281 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106285 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106288 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106290 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106301 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106304 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106306 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106309 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106313 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106317 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106322 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106325 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106328 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106331 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:51.116361 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106334 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106337 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106340 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106343 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106346 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106349 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106352 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106355 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106358 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106361 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106363 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106366 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106369 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106371 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106374 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106377 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106379 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106383 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106385 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106390 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:51.116979 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106392 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106395 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106398 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106400 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106414 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106417 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106420 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106422 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106426 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106429 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106431 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106434 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106437 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106439 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106442 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106445 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106447 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106450 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106453 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106456 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:51.117806 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106459 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106461 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106464 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106467 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106469 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106472 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106474 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106477 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106480 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106483 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106485 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106489 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106491 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106494 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106497 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106499 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106502 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106506 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106510 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:51.118364 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106513 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106517 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106520 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.106522 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.106527 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.114005 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.114023 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.114799 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.114817 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114905 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114913 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114918 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114923 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114930 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:51.118895 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114936 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114941 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114946 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114951 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114956 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114961 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114966 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114971 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114975 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114980 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114984 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114989 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114993 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.114996 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115001 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115006 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115011 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115015 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115019 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:51.119246 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115023 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115028 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115032 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115036 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115040 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115046 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115051 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115056 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115060 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115064 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115068 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115072 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115077 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115081 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115085 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115089 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115093 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115098 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115102 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115106 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:51.119775 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115111 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115115 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115119 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115123 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115127 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115131 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115135 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115140 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115144 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115148 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115152 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115156 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115163 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115169 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115175 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115179 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115183 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115188 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115193 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115198 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:51.120265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115202 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115206 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115210 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115214 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115219 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115223 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115228 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115232 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115236 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115240 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115244 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115249 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115253 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115257 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115262 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115266 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115270 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115277 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115282 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:51.120787 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115287 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115291 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115295 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115300 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115304 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115309 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115313 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115317 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115321 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115326 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115332 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115336 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115340 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115345 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115349 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:51.121254 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115354 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.115362 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115547 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115556 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115560 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115565 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115570 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115576 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115582 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115587 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115592 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115597 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115601 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115606 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 20:36:51.121640 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115610 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115614 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115618 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115623 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115627 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115631 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115635 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115639 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115644 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115648 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115652 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115657 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115661 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115666 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115670 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115675 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115680 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115685 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115689 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 20:36:51.122010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115694 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115698 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115702 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115706 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115710 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115714 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115718 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115722 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115726 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115730 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115735 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115739 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115743 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115749 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115754 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115759 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115763 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115767 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115771 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115775 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 20:36:51.122485 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115779 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115783 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115787 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115791 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115796 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115800 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115804 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115809 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115813 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115817 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115822 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115826 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115830 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115835 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115839 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115844 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115848 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115852 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115856 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115860 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 20:36:51.122976 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115865 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115869 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115873 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115877 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115881 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115885 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115889 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115893 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115898 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115902 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115907 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115911 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115915 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115919 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115923 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115927 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115931 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115938 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115943 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 20:36:51.123481 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115948 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115953 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115958 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115962 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115967 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115971 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115975 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115979 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:51.115983 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.115991 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.116125 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.119332 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.120227 2580 server.go:1019] "Starting client certificate rotation" Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.120320 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 20:36:51.123970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.120350 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 20:36:51.124321 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.123623 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 20:36:51.126150 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.126134 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 20:36:51.137285 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.137265 2580 log.go:25] "Validated CRI v1 runtime API" Sep 15 20:36:51.140024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.140010 2580 log.go:25] "Validated CRI v1 image API" Sep 15 20:36:51.140873 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.140858 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 20:36:51.143482 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.143463 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7f201a18-ec06-4289-94ee-23e75a67d816:/dev/nvme0n1p4 e0dbb1c5-7b3b-46f7-8858-3303e4935c09:/dev/nvme0n1p3] Sep 15 20:36:51.143527 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.143483 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 20:36:51.145432 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.145401 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 20:36:51.149274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.149160 2580 manager.go:216] Machine: {Timestamp:2026-09-15 20:36:51.14799919 +0000 UTC m=+0.277735763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099882 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27753b217c9e2e0b4699e88c417877 SystemUUID:ec27753b-217c-9e2e-0b46-99e88c417877 BootID:b6f72e66-80ff-4591-a4b6-9b4fc5b129e3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:43:64:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:43:64:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:c8:57:17:53:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 20:36:51.149274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.149269 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 20:36:51.149384 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.149348 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 20:36:51.151769 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.151736 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 20:36:51.151901 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.151771 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 20:36:51.152466 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.152457 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 20:36:51.152503 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.152468 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 20:36:51.152503 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.152475 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 20:36:51.153549 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.153539 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 20:36:51.154358 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.154348 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 20:36:51.158425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158413 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 15 20:36:51.158535 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158525 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 20:36:51.158572 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158558 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 15 20:36:51.158600 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158573 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 20:36:51.158600 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158587 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 20:36:51.158655 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158602 2580 kubelet.go:398] "Adding apiserver pod source" Sep 15 20:36:51.158655 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158616 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 20:36:51.158952 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158940 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 20:36:51.159017 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.158961 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 20:36:51.162849 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.162825 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 20:36:51.165909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.164856 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqj4g" Sep 15 20:36:51.166088 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.166072 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 20:36:51.166159 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.166105 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 20:36:51.167672 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167660 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167676 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167683 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167688 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167696 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167702 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167715 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167721 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167727 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 20:36:51.167731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167732 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 20:36:51.167972 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167741 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 20:36:51.167972 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.167746 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 20:36:51.168389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.168379 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 20:36:51.168389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.168387 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 20:36:51.171320 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.171301 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 20:36:51.171996 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.171972 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 20:36:51.172080 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.172046 2580 server.go:1268] "Started kubelet" Sep 15 20:36:51.172080 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.172043 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 20:36:51.172153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.172100 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 20:36:51.172185 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.172128 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 20:36:51.172216 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.172197 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 20:36:51.173147 ip-10-0-139-201 systemd[1]: Started Kubernetes Kubelet. Sep 15 20:36:51.174167 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.173198 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 20:36:51.174167 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.174070 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqj4g" Sep 15 20:36:51.174544 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.174473 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 15 20:36:51.176212 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.176196 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 20:36:51.179297 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.178505 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-201.ec2.internal.18d598f670d4f485 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-201.ec2.internal,UID:ip-10-0-139-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-201.ec2.internal,},FirstTimestamp:2026-09-15 20:36:51.172021381 +0000 UTC m=+0.301757955,LastTimestamp:2026-09-15 20:36:51.172021381 +0000 UTC m=+0.301757955,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-201.ec2.internal,}" Sep 15 20:36:51.179623 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.179604 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 20:36:51.180125 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.180110 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 20:36:51.180220 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.180200 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.180835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.180801 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 20:36:51.180911 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.180839 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 20:36:51.180911 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.180849 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 20:36:51.181010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.180993 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 15 20:36:51.181010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.181001 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 15 20:36:51.181948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.181912 2580 factory.go:55] Registering systemd factory Sep 15 20:36:51.182032 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.181963 2580 factory.go:223] Registration of the systemd container factory successfully Sep 15 20:36:51.182549 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.182529 2580 factory.go:153] Registering CRI-O factory Sep 15 20:36:51.182549 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.182551 2580 factory.go:223] Registration of the crio container factory successfully Sep 15 20:36:51.182690 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.182637 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 20:36:51.182690 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.182658 2580 factory.go:103] Registering Raw factory Sep 15 20:36:51.182690 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.182674 2580 manager.go:1196] Started watching for new ooms in manager Sep 15 20:36:51.183191 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.183168 2580 manager.go:349] Starting recovery of all containers Sep 15 20:36:51.183802 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.183783 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:51.187415 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.187376 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.197554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.197354 2580 manager.go:354] Recovery completed Sep 15 20:36:51.199096 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.199065 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 15 20:36:51.202337 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.202323 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:51.205461 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.205440 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:51.205545 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.205473 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:51.205545 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.205489 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:51.206176 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.206155 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 20:36:51.206176 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.206170 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 20:36:51.206291 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.206187 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:51.206291 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.206191 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 15 20:36:51.207003 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.206987 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:51.207066 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.207018 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:51.207066 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.207032 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:51.208450 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.208438 2580 policy_none.go:49] "None policy: Start" Sep 15 20:36:51.208498 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.208455 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 20:36:51.208498 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.208464 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 20:36:51.210546 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.210526 2580 policy_none.go:47] "Start" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.245940 2580 manager.go:337] "Starting Device Plugin manager" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.245972 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.245981 2580 server.go:87] "Starting device plugin registration server" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.246330 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.246343 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.246443 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.246528 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.246537 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.247140 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 20:36:51.256370 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.247197 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.305896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.305862 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 20:36:51.307020 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.307002 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 20:36:51.307104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.307047 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 20:36:51.307104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.307070 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 20:36:51.307205 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.307161 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 20:36:51.309211 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.309185 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:51.346969 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.346909 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:51.347796 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.347782 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:51.347856 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.347810 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:51.347856 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.347820 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:51.347856 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.347841 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.355260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.355244 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.355326 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.355266 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-201.ec2.internal\": node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.369129 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.369107 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.407552 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.407522 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"] Sep 15 20:36:51.407636 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.407590 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:51.408471 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.408457 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:51.408541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.408485 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:51.408541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.408500 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:51.410932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.410918 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:51.411078 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.411062 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.411194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.411103 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:51.412941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.412925 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:51.413030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.412950 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:51.413030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.412959 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:51.413030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.412971 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:51.413030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.412977 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:51.413030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.412988 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:51.415304 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.415288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.415383 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.415317 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 20:36:51.416949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.416929 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 15 20:36:51.417030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.416960 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 20:36:51.417030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.416975 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 15 20:36:51.444234 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.444216 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.448785 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.448768 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.469733 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.469712 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.482479 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.482465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.482546 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.482486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.482546 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.482504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b23ec63d9b3d220644ea926d481ee00f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"b23ec63d9b3d220644ea926d481ee00f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.570823 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.570804 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.583127 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.583110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.583183 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.583134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.583183 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.583156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b23ec63d9b3d220644ea926d481ee00f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"b23ec63d9b3d220644ea926d481ee00f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.583247 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.583180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b23ec63d9b3d220644ea926d481ee00f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"b23ec63d9b3d220644ea926d481ee00f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.583247 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.583210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.583247 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.583217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.671555 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.671491 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.747010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.746984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.752090 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:51.752077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 15 20:36:51.772156 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.772124 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.872662 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.872631 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:51.973187 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:51.973133 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.073597 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:52.073574 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.120064 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.120046 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 20:36:52.120453 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.120168 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 20:36:52.120453 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.120202 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 20:36:52.174505 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:52.174483 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.180533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.180498 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 20:31:52 +0000 UTC" deadline="2028-06-04 15:02:22.31288364 +0000 UTC" Sep 15 20:36:52.180533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.180527 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15066h25m30.132359705s" Sep 15 20:36:52.275586 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:52.275549 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.277542 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:52.277507 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef12fc99af254fa0de9a3b0e067caf7.slice/crio-bd13d84ecfece174550d3681683c52e7d683407c19fe1f5c9f5aa8a71db174a5 WatchSource:0}: Error finding container bd13d84ecfece174550d3681683c52e7d683407c19fe1f5c9f5aa8a71db174a5: Status 404 returned error can't find the container with id bd13d84ecfece174550d3681683c52e7d683407c19fe1f5c9f5aa8a71db174a5 Sep 15 20:36:52.281085 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:52.281061 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23ec63d9b3d220644ea926d481ee00f.slice/crio-ce44d74e1a4bb2f7d0d6c394e90e0de0ac825723e69e5311d3e420bdf3eaec90 WatchSource:0}: Error finding container ce44d74e1a4bb2f7d0d6c394e90e0de0ac825723e69e5311d3e420bdf3eaec90: Status 404 returned error can't find the container with id ce44d74e1a4bb2f7d0d6c394e90e0de0ac825723e69e5311d3e420bdf3eaec90 Sep 15 20:36:52.283708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.283692 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:36:52.311464 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.311398 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"cef12fc99af254fa0de9a3b0e067caf7","Type":"ContainerStarted","Data":"bd13d84ecfece174550d3681683c52e7d683407c19fe1f5c9f5aa8a71db174a5"} Sep 15 20:36:52.312648 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.312619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"b23ec63d9b3d220644ea926d481ee00f","Type":"ContainerStarted","Data":"ce44d74e1a4bb2f7d0d6c394e90e0de0ac825723e69e5311d3e420bdf3eaec90"} Sep 15 20:36:52.377238 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:52.377200 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.483549 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:52.483523 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.578501 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.577162 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:52.584512 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:52.584490 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.685603 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:52.685578 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 15 20:36:52.754074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.754044 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:52.780740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.780705 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 15 20:36:52.792361 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.792342 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 20:36:52.793637 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.793614 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 15 20:36:52.802881 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:52.802856 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 20:36:53.139771 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.139737 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:53.159606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.159576 2580 apiserver.go:52] "Watching apiserver" Sep 15 20:36:53.166762 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.166737 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 20:36:53.168540 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.168511 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-pr5lj","openshift-dns/node-resolver-2ssxn","openshift-multus/network-metrics-daemon-5vvbb","openshift-network-diagnostics/network-check-target-7gk2l","openshift-ovn-kubernetes/ovnkube-node-bzkb9","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x","openshift-image-registry/node-ca-vjcfq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","openshift-multus/multus-additional-cni-plugins-q4jtq","openshift-multus/multus-ckvww","openshift-network-operator/iptables-alerter-4w2w2","kube-system/global-pull-secret-syncer-6v4bn","kube-system/konnectivity-agent-7tjzq"] Sep 15 20:36:53.171068 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.171046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.173499 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.173460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.173618 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.173546 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:36:53.173693 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.173646 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 20:36:53.173867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.173841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 20:36:53.173938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.173882 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsrfz\"" Sep 15 20:36:53.174253 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.174229 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 20:36:53.175975 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.175953 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:53.176081 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.176028 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:36:53.178297 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.178274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:53.178403 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.178347 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:36:53.180052 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.180032 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 20:36:53.180881 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.180860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.183241 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183216 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-69mt2\"" Sep 15 20:36:53.183436 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183393 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.183591 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 20:36:53.183703 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183606 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 20:36:53.183773 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183756 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 20:36:53.183813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183781 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 20:36:53.183906 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183884 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 20:36:53.184004 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.183986 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 20:36:53.185681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.185656 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 20:36:53.186512 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.186202 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 20:36:53.186512 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.186333 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6zwwd\"" Sep 15 20:36:53.186661 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.186571 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.188513 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.188493 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 20:36:53.189115 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.188749 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 20:36:53.189115 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.188834 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 20:36:53.189115 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.188961 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4p89d\"" Sep 15 20:36:53.189115 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.189080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.191458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191329 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 20:36:53.191458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191357 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6cc2t\"" Sep 15 20:36:53.191458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73b7914e-e824-44f9-8a97-d735b8de33a7-dbus\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.191458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191410 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:36:53.191458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ls4w\" (UniqueName: \"kubernetes.io/projected/dad0237e-3438-4e30-a3f8-5f76d25d0763-kube-api-access-8ls4w\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:53.191458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-systemd-units\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.191839 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97adad06-f0b5-4467-b20f-9dafdc3058a6-agent-certs\") pod \"konnectivity-agent-7tjzq\" (UID: \"97adad06-f0b5-4467-b20f-9dafdc3058a6\") " pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.191839 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.191839 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:53.191839 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-run-netns\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.191839 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-node-log\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.191839 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-socket-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.191839 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73b7914e-e824-44f9-8a97-d735b8de33a7-kubelet-config\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.192171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:53.192171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-kubelet\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.192171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-slash\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.192171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-systemd\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.192171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.191984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-etc-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.192171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.192002 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.195977 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.195918 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196471 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kqk2l\"" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.192000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovn-node-metrics-cert\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196522 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovnkube-script-lib\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxqkw\" (UniqueName: \"kubernetes.io/projected/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-kube-api-access-jxqkw\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-device-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-cni-netd\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-sys-fs\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-cni-bin\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-env-overrides\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.196982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-var-lib-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-ovn\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-log-socket\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.197508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97adad06-f0b5-4467-b20f-9dafdc3058a6-konnectivity-ca\") pod \"konnectivity-agent-7tjzq\" (UID: \"97adad06-f0b5-4467-b20f-9dafdc3058a6\") " pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.198374 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.198374 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-registration-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.198374 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.198374 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovnkube-config\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.198374 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.198374 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.197411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jqn\" (UniqueName: \"kubernetes.io/projected/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-kube-api-access-j2jqn\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.199625 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.199605 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 20:36:53.199819 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.199744 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 20:36:53.200591 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.200566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.200701 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.200672 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 20:36:53.200891 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.200872 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 20:36:53.200966 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.200923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 20:36:53.201310 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.201118 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vpqft\"" Sep 15 20:36:53.201310 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.201173 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 20:36:53.203287 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.203119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.203287 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.203238 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 20:36:53.203287 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.203254 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ps6zw\"" Sep 15 20:36:53.205835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.205814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmvl2\"" Sep 15 20:36:53.205940 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.205890 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 20:36:53.206168 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.206143 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 20:36:53.206397 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.206378 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:36:53.226428 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.226407 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hv4z" Sep 15 20:36:53.237825 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.237801 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hv4z" Sep 15 20:36:53.276136 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.276111 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 20:36:53.282808 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.282786 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 20:36:53.298265 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gwb\" (UniqueName: \"kubernetes.io/projected/43127936-57a8-463b-ac1b-2568e6fa2347-kube-api-access-98gwb\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.298375 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-systemd-units\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.298375 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:53.298375 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.298569 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-run-netns\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.298814 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298789 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-systemd-units\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.298814 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-run-netns\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.298943 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-socket-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.298943 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:53.298943 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-systemd\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.299092 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a1999b9-32f4-4058-aef0-078fbd9ca031-host-slash\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.299092 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-socket-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.299092 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.298973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-kubelet\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299092 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovn-node-metrics-cert\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299092 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.299051 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:53.299336 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.299126 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:53.799103072 +0000 UTC m=+2.928839663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:53.299336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-device-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.299336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-sys-fs\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.299336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgn9\" (UniqueName: \"kubernetes.io/projected/a11c1990-58c3-4921-84c7-f7bf86deaddf-kube-api-access-rqgn9\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.299336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-tuned\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.299336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.299336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-k8s-cni-cncf-io\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-kubelet\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-cni-netd\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299393 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-host\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-tmp\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-system-cni-dir\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-cni-netd\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-device-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-cnibin\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-sys-fs\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d6afa5-55c1-484f-9539-b6d240fdf155-cni-binary-copy\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-conf-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-env-overrides\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.299740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299820 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-multus-certs\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-var-lib-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.299983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-ovn\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z5l5\" (UniqueName: \"kubernetes.io/projected/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-kube-api-access-2z5l5\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8qc\" (UniqueName: \"kubernetes.io/projected/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-kube-api-access-2b8qc\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300070 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-system-cni-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-daemon-config\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-log-socket\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-env-overrides\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-lib-modules\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-cnibin\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-etc-kubernetes\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysconfig\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.300521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovnkube-config\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jqn\" (UniqueName: \"kubernetes.io/projected/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-kube-api-access-j2jqn\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ls4w\" (UniqueName: \"kubernetes.io/projected/dad0237e-3438-4e30-a3f8-5f76d25d0763-kube-api-access-8ls4w\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a11c1990-58c3-4921-84c7-f7bf86deaddf-serviceca\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-socket-dir-parent\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-cni-bin\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-log-socket\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-kubelet\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97adad06-f0b5-4467-b20f-9dafdc3058a6-agent-certs\") pod \"konnectivity-agent-7tjzq\" (UID: \"97adad06-f0b5-4467-b20f-9dafdc3058a6\") " pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-modprobe-d\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-var-lib-kubelet\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-netns\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-node-log\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-var-lib-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73b7914e-e824-44f9-8a97-d735b8de33a7-kubelet-config\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.301376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.300998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-kubernetes\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-run\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-sys\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301035 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-ovn\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovnkube-config\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-node-log\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.301274 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73b7914e-e824-44f9-8a97-d735b8de33a7-kubelet-config\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.301323 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret podName:73b7914e-e824-44f9-8a97-d735b8de33a7 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:53.801306927 +0000 UTC m=+2.931043494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret") pod "global-pull-secret-syncer-6v4bn" (UID: "73b7914e-e824-44f9-8a97-d735b8de33a7") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-slash\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-slash\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-systemd\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-etc-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-systemd\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovnkube-script-lib\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-etc-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.302308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxqkw\" (UniqueName: \"kubernetes.io/projected/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-kube-api-access-jxqkw\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-hostroot\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysctl-d\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-cni-bin\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.301903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-hosts-file\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-host-cni-bin\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-cni-multus\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797sf\" (UniqueName: \"kubernetes.io/projected/b2d6afa5-55c1-484f-9539-b6d240fdf155-kube-api-access-797sf\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6hb\" (UniqueName: \"kubernetes.io/projected/4a1999b9-32f4-4058-aef0-078fbd9ca031-kube-api-access-4j6hb\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-tmp-dir\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97adad06-f0b5-4467-b20f-9dafdc3058a6-konnectivity-ca\") pod \"konnectivity-agent-7tjzq\" (UID: \"97adad06-f0b5-4467-b20f-9dafdc3058a6\") " pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a11c1990-58c3-4921-84c7-f7bf86deaddf-host\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovnkube-script-lib\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-cni-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysctl-conf\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a1999b9-32f4-4058-aef0-078fbd9ca031-iptables-alerter-script\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.303179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-registration-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73b7914e-e824-44f9-8a97-d735b8de33a7-dbus\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-registration-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73b7914e-e824-44f9-8a97-d735b8de33a7-dbus\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-run-openvswitch\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.303964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.302809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97adad06-f0b5-4467-b20f-9dafdc3058a6-konnectivity-ca\") pod \"konnectivity-agent-7tjzq\" (UID: \"97adad06-f0b5-4467-b20f-9dafdc3058a6\") " pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.305026 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.304958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-ovn-node-metrics-cert\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.305637 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.305520 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:53.305637 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.305546 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:53.305637 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.305560 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nqxlg for pod openshift-network-diagnostics/network-check-target-7gk2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:53.305637 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.305614 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg podName:4aa4af63-23fe-4afd-ba45-8d1926913384 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:53.805597805 +0000 UTC m=+2.935334372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nqxlg" (UniqueName: "kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg") pod "network-check-target-7gk2l" (UID: "4aa4af63-23fe-4afd-ba45-8d1926913384") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:53.307133 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.307109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97adad06-f0b5-4467-b20f-9dafdc3058a6-agent-certs\") pod \"konnectivity-agent-7tjzq\" (UID: \"97adad06-f0b5-4467-b20f-9dafdc3058a6\") " pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.316426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.315994 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jqn\" (UniqueName: \"kubernetes.io/projected/4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3-kube-api-access-j2jqn\") pod \"aws-ebs-csi-driver-node-7wx6x\" (UID: \"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.316426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.316219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxqkw\" (UniqueName: \"kubernetes.io/projected/0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397-kube-api-access-jxqkw\") pod \"ovnkube-node-bzkb9\" (UID: \"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.316426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.316235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ls4w\" (UniqueName: \"kubernetes.io/projected/dad0237e-3438-4e30-a3f8-5f76d25d0763-kube-api-access-8ls4w\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:53.403818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.403722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-systemd\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.403818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.403767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a1999b9-32f4-4058-aef0-078fbd9ca031-host-slash\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.403818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.403796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgn9\" (UniqueName: \"kubernetes.io/projected/a11c1990-58c3-4921-84c7-f7bf86deaddf-kube-api-access-rqgn9\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.404118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.403821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-tuned\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.404118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.403944 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-systemd\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.404118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.404118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-k8s-cni-cncf-io\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.403956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a1999b9-32f4-4058-aef0-078fbd9ca031-host-slash\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.404118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-host\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.404118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-k8s-cni-cncf-io\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-host\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-tmp\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-system-cni-dir\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-cnibin\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d6afa5-55c1-484f-9539-b6d240fdf155-cni-binary-copy\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-conf-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-multus-certs\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z5l5\" (UniqueName: \"kubernetes.io/projected/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-kube-api-access-2z5l5\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8qc\" (UniqueName: \"kubernetes.io/projected/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-kube-api-access-2b8qc\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-cnibin\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-system-cni-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-daemon-config\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404391 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-system-cni-dir\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-lib-modules\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-cnibin\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.404486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-etc-kubernetes\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysconfig\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a11c1990-58c3-4921-84c7-f7bf86deaddf-serviceca\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-socket-dir-parent\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-conf-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-cni-bin\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-cni-bin\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-multus-certs\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.404787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-system-cni-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405032 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-etc-kubernetes\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-lib-modules\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-cnibin\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-daemon-config\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405267 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2d6afa5-55c1-484f-9539-b6d240fdf155-cni-binary-copy\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-kubelet\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-modprobe-d\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.405392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-var-lib-kubelet\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405381 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-kubelet\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-socket-dir-parent\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-netns\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405426 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysconfig\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-kubernetes\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-run\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-modprobe-d\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-run-netns\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-kubernetes\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405605 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-run\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-var-lib-kubelet\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-sys\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-hostroot\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405683 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a11c1990-58c3-4921-84c7-f7bf86deaddf-serviceca\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysctl-d\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-sys\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-hosts-file\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.406240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-hostroot\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-cni-multus\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797sf\" (UniqueName: \"kubernetes.io/projected/b2d6afa5-55c1-484f-9539-b6d240fdf155-kube-api-access-797sf\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-hosts-file\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6hb\" (UniqueName: \"kubernetes.io/projected/4a1999b9-32f4-4058-aef0-078fbd9ca031-kube-api-access-4j6hb\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysctl-d\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405805 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-host-var-lib-cni-multus\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-tmp-dir\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a11c1990-58c3-4921-84c7-f7bf86deaddf-host\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-cni-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysctl-conf\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a11c1990-58c3-4921-84c7-f7bf86deaddf-host\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a1999b9-32f4-4058-aef0-078fbd9ca031-iptables-alerter-script\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.405978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gwb\" (UniqueName: \"kubernetes.io/projected/43127936-57a8-463b-ac1b-2568e6fa2347-kube-api-access-98gwb\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43127936-57a8-463b-ac1b-2568e6fa2347-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.407069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2d6afa5-55c1-484f-9539-b6d240fdf155-multus-cni-dir\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.407813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-tmp-dir\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.407813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-sysctl-conf\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.407813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43127936-57a8-463b-ac1b-2568e6fa2347-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.407813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-etc-tuned\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.407813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.406838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a1999b9-32f4-4058-aef0-078fbd9ca031-iptables-alerter-script\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.408276 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.408248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-tmp\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.416937 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.416761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8qc\" (UniqueName: \"kubernetes.io/projected/0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809-kube-api-access-2b8qc\") pod \"node-resolver-2ssxn\" (UID: \"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809\") " pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.416937 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.416852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gwb\" (UniqueName: \"kubernetes.io/projected/43127936-57a8-463b-ac1b-2568e6fa2347-kube-api-access-98gwb\") pod \"multus-additional-cni-plugins-q4jtq\" (UID: \"43127936-57a8-463b-ac1b-2568e6fa2347\") " pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.417458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.417437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6hb\" (UniqueName: \"kubernetes.io/projected/4a1999b9-32f4-4058-aef0-078fbd9ca031-kube-api-access-4j6hb\") pod \"iptables-alerter-4w2w2\" (UID: \"4a1999b9-32f4-4058-aef0-078fbd9ca031\") " pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.417711 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.417688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z5l5\" (UniqueName: \"kubernetes.io/projected/785472ab-8c31-4ca5-bfd7-07b33eeaa1b2-kube-api-access-2z5l5\") pod \"tuned-pr5lj\" (UID: \"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2\") " pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.419471 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.419444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgn9\" (UniqueName: \"kubernetes.io/projected/a11c1990-58c3-4921-84c7-f7bf86deaddf-kube-api-access-rqgn9\") pod \"node-ca-vjcfq\" (UID: \"a11c1990-58c3-4921-84c7-f7bf86deaddf\") " pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.419615 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.419537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-797sf\" (UniqueName: \"kubernetes.io/projected/b2d6afa5-55c1-484f-9539-b6d240fdf155-kube-api-access-797sf\") pod \"multus-ckvww\" (UID: \"b2d6afa5-55c1-484f-9539-b6d240fdf155\") " pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.483567 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.483527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" Sep 15 20:36:53.496759 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.496731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:36:53.507667 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.507645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:36:53.514443 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.514422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vjcfq" Sep 15 20:36:53.523240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.523221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" Sep 15 20:36:53.529004 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.528986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ssxn" Sep 15 20:36:53.536751 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.536734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" Sep 15 20:36:53.543676 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.543659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ckvww" Sep 15 20:36:53.551554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.551536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4w2w2" Sep 15 20:36:53.808662 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.808623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:53.808853 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.808693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:53.808853 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:53.808729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:53.808853 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808800 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:53.808853 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808821 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:53.808853 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808835 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:53.808853 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808842 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:53.808853 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808856 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nqxlg for pod openshift-network-diagnostics/network-check-target-7gk2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:53.809173 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808886 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret podName:73b7914e-e824-44f9-8a97-d735b8de33a7 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:54.808865413 +0000 UTC m=+3.938602001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret") pod "global-pull-secret-syncer-6v4bn" (UID: "73b7914e-e824-44f9-8a97-d735b8de33a7") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:53.809173 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808904 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:54.808896789 +0000 UTC m=+3.938633353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:53.809173 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:53.808920 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg podName:4aa4af63-23fe-4afd-ba45-8d1926913384 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:54.808913335 +0000 UTC m=+3.938649897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqxlg" (UniqueName: "kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg") pod "network-check-target-7gk2l" (UID: "4aa4af63-23fe-4afd-ba45-8d1926913384") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:54.242254 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:54.242157 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:31:53 +0000 UTC" deadline="2028-04-28 10:24:24.35609817 +0000 UTC" Sep 15 20:36:54.242254 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:54.242198 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14173h47m30.113904621s" Sep 15 20:36:54.333316 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.333096 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11c1990_58c3_4921_84c7_f7bf86deaddf.slice/crio-76d4fd10868d712619ca16d3da470f7e6d154c9b71f17abb6797d5c712052e7d WatchSource:0}: Error finding container 76d4fd10868d712619ca16d3da470f7e6d154c9b71f17abb6797d5c712052e7d: Status 404 returned error can't find the container with id 76d4fd10868d712619ca16d3da470f7e6d154c9b71f17abb6797d5c712052e7d Sep 15 20:36:54.334948 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.334652 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785472ab_8c31_4ca5_bfd7_07b33eeaa1b2.slice/crio-5b2c06430d7cb7f851ba4e3b0779d8362a94564d06f657d990b060983028c625 WatchSource:0}: Error finding container 5b2c06430d7cb7f851ba4e3b0779d8362a94564d06f657d990b060983028c625: Status 404 returned error can't find the container with id 5b2c06430d7cb7f851ba4e3b0779d8362a94564d06f657d990b060983028c625 Sep 15 20:36:54.337726 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.337682 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97adad06_f0b5_4467_b20f_9dafdc3058a6.slice/crio-ff9f690ca332ee7f173f01ae5fd9720a648ed6638eab2eb7d044a71bc9f8ce1e WatchSource:0}: Error finding container ff9f690ca332ee7f173f01ae5fd9720a648ed6638eab2eb7d044a71bc9f8ce1e: Status 404 returned error can't find the container with id ff9f690ca332ee7f173f01ae5fd9720a648ed6638eab2eb7d044a71bc9f8ce1e Sep 15 20:36:54.339858 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.339815 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43127936_57a8_463b_ac1b_2568e6fa2347.slice/crio-a9f473f8a8f9ea981daed79f304ec6a0e69b369f94e7242608fbc77108946ac9 WatchSource:0}: Error finding container a9f473f8a8f9ea981daed79f304ec6a0e69b369f94e7242608fbc77108946ac9: Status 404 returned error can't find the container with id a9f473f8a8f9ea981daed79f304ec6a0e69b369f94e7242608fbc77108946ac9 Sep 15 20:36:54.341201 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.340954 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecf3d66_c8a6_4ecb_b83d_ddb7e8db3809.slice/crio-e9daa73adac80d6fe48d78a0e3d80749e563c1238c5db8ee957efde7f49d5bea WatchSource:0}: Error finding container e9daa73adac80d6fe48d78a0e3d80749e563c1238c5db8ee957efde7f49d5bea: Status 404 returned error can't find the container with id e9daa73adac80d6fe48d78a0e3d80749e563c1238c5db8ee957efde7f49d5bea Sep 15 20:36:54.342232 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.342209 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d6afa5_55c1_484f_9539_b6d240fdf155.slice/crio-59259ea8a57c350305f6d24793f286b1ef1b9fa9e351670f11cbefc2996895cd WatchSource:0}: Error finding container 59259ea8a57c350305f6d24793f286b1ef1b9fa9e351670f11cbefc2996895cd: Status 404 returned error can't find the container with id 59259ea8a57c350305f6d24793f286b1ef1b9fa9e351670f11cbefc2996895cd Sep 15 20:36:54.344784 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.344727 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0aef44_c4ae_42a0_aa65_0abbf2c0b1b3.slice/crio-97553c48947ed88af648fdb30ba0705323e9456b2d6261eec13906c37865fcab WatchSource:0}: Error finding container 97553c48947ed88af648fdb30ba0705323e9456b2d6261eec13906c37865fcab: Status 404 returned error can't find the container with id 97553c48947ed88af648fdb30ba0705323e9456b2d6261eec13906c37865fcab Sep 15 20:36:54.345667 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.345642 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1999b9_32f4_4058_aef0_078fbd9ca031.slice/crio-5ac77259e4de3cdd2852a2fcab819d102fed62728d82b053985610b8e27b5ce3 WatchSource:0}: Error finding container 5ac77259e4de3cdd2852a2fcab819d102fed62728d82b053985610b8e27b5ce3: Status 404 returned error can't find the container with id 5ac77259e4de3cdd2852a2fcab819d102fed62728d82b053985610b8e27b5ce3 Sep 15 20:36:54.347319 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:36:54.347295 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fad074d_4b1c_4a3e_aff3_d9a0fd4d1397.slice/crio-c9a6ba397a65e0340484a8df45bbb22250ffce618bf42621222140124704f60b WatchSource:0}: Error finding container c9a6ba397a65e0340484a8df45bbb22250ffce618bf42621222140124704f60b: Status 404 returned error can't find the container with id c9a6ba397a65e0340484a8df45bbb22250ffce618bf42621222140124704f60b Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:54.817670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:54.817873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:54.817911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818046 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818125 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:56.818106286 +0000 UTC m=+5.947842853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818200 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818237 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret podName:73b7914e-e824-44f9-8a97-d735b8de33a7 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:56.818224893 +0000 UTC m=+5.947961473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret") pod "global-pull-secret-syncer-6v4bn" (UID: "73b7914e-e824-44f9-8a97-d735b8de33a7") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818328 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818344 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818358 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nqxlg for pod openshift-network-diagnostics/network-check-target-7gk2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:54.818856 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:54.818397 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg podName:4aa4af63-23fe-4afd-ba45-8d1926913384 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:56.818384842 +0000 UTC m=+5.948121409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqxlg" (UniqueName: "kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg") pod "network-check-target-7gk2l" (UID: "4aa4af63-23fe-4afd-ba45-8d1926913384") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:55.242656 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.242354 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:31:53 +0000 UTC" deadline="2028-02-19 09:25:30.815565507 +0000 UTC" Sep 15 20:36:55.242656 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.242431 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12516h48m35.573138893s" Sep 15 20:36:55.309655 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.309622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:55.309828 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.309641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:55.309828 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:55.309772 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:36:55.309828 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:55.309812 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:36:55.310161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.309867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:55.310161 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:55.310006 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:36:55.326514 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.326481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ckvww" event={"ID":"b2d6afa5-55c1-484f-9539-b6d240fdf155","Type":"ContainerStarted","Data":"59259ea8a57c350305f6d24793f286b1ef1b9fa9e351670f11cbefc2996895cd"} Sep 15 20:36:55.335676 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.335644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4w2w2" event={"ID":"4a1999b9-32f4-4058-aef0-078fbd9ca031","Type":"ContainerStarted","Data":"5ac77259e4de3cdd2852a2fcab819d102fed62728d82b053985610b8e27b5ce3"} Sep 15 20:36:55.343585 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.343493 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" event={"ID":"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3","Type":"ContainerStarted","Data":"97553c48947ed88af648fdb30ba0705323e9456b2d6261eec13906c37865fcab"} Sep 15 20:36:55.348758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.348730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ssxn" event={"ID":"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809","Type":"ContainerStarted","Data":"e9daa73adac80d6fe48d78a0e3d80749e563c1238c5db8ee957efde7f49d5bea"} Sep 15 20:36:55.353900 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.353871 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerStarted","Data":"a9f473f8a8f9ea981daed79f304ec6a0e69b369f94e7242608fbc77108946ac9"} Sep 15 20:36:55.357778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.357750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tjzq" event={"ID":"97adad06-f0b5-4467-b20f-9dafdc3058a6","Type":"ContainerStarted","Data":"ff9f690ca332ee7f173f01ae5fd9720a648ed6638eab2eb7d044a71bc9f8ce1e"} Sep 15 20:36:55.365659 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.365434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" event={"ID":"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2","Type":"ContainerStarted","Data":"5b2c06430d7cb7f851ba4e3b0779d8362a94564d06f657d990b060983028c625"} Sep 15 20:36:55.377528 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.377403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"b23ec63d9b3d220644ea926d481ee00f","Type":"ContainerStarted","Data":"6abf64ce7d55adb02925fce835b7366adf8d1a6de2a80b2dc44f0cbc2c5cc185"} Sep 15 20:36:55.394223 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.394190 2580 generic.go:356] "Generic (PLEG): container finished" podID="cef12fc99af254fa0de9a3b0e067caf7" containerID="184df5b903906690cee19a1db97f22da92a93f12a261c2e16bba124847babd54" exitCode=0 Sep 15 20:36:55.394407 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.394275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"cef12fc99af254fa0de9a3b0e067caf7","Type":"ContainerDied","Data":"184df5b903906690cee19a1db97f22da92a93f12a261c2e16bba124847babd54"} Sep 15 20:36:55.406350 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.406121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"c9a6ba397a65e0340484a8df45bbb22250ffce618bf42621222140124704f60b"} Sep 15 20:36:55.411419 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.410900 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" podStartSLOduration=2.41088131 podStartE2EDuration="2.41088131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:36:55.395017132 +0000 UTC m=+4.524753721" watchObservedRunningTime="2026-09-15 20:36:55.41088131 +0000 UTC m=+4.540617898" Sep 15 20:36:55.413432 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:55.413386 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vjcfq" event={"ID":"a11c1990-58c3-4921-84c7-f7bf86deaddf","Type":"ContainerStarted","Data":"76d4fd10868d712619ca16d3da470f7e6d154c9b71f17abb6797d5c712052e7d"} Sep 15 20:36:56.436757 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:56.436539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"cef12fc99af254fa0de9a3b0e067caf7","Type":"ContainerStarted","Data":"003d2dfc69ce0b5077b320094b0dd56003ae4a2e2b39ebc474112d6e0cccf8bd"} Sep 15 20:36:56.840894 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:56.840819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:56.841163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:56.840917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:56.841163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:56.840983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:56.841163 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841125 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:56.841368 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841192 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret podName:73b7914e-e824-44f9-8a97-d735b8de33a7 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:00.841172117 +0000 UTC m=+9.970908705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret") pod "global-pull-secret-syncer-6v4bn" (UID: "73b7914e-e824-44f9-8a97-d735b8de33a7") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:36:56.841675 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841652 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:36:56.841786 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841681 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:36:56.841786 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841694 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nqxlg for pod openshift-network-diagnostics/network-check-target-7gk2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:56.841786 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841747 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg podName:4aa4af63-23fe-4afd-ba45-8d1926913384 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:00.841729679 +0000 UTC m=+9.971466257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqxlg" (UniqueName: "kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg") pod "network-check-target-7gk2l" (UID: "4aa4af63-23fe-4afd-ba45-8d1926913384") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:36:56.842161 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841834 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:56.842161 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:56.841871 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:00.841857812 +0000 UTC m=+9.971594389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:36:57.308620 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:57.308558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:57.308783 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:57.308686 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:36:57.309083 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:57.309060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:57.309210 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:57.309184 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:36:57.310573 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:57.310552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:57.310686 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:57.310649 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:36:59.308016 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:59.307916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:36:59.308016 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:59.307948 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:36:59.309101 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:59.308103 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:36:59.309101 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:59.308583 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:36:59.309101 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:36:59.308668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:36:59.309101 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:36:59.308778 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:00.876184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:00.876150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:00.876184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:00.876188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:00.876221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876335 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876395 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret podName:73b7914e-e824-44f9-8a97-d735b8de33a7 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:08.876374897 +0000 UTC m=+18.006111466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret") pod "global-pull-secret-syncer-6v4bn" (UID: "73b7914e-e824-44f9-8a97-d735b8de33a7") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876507 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876522 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876535 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nqxlg for pod openshift-network-diagnostics/network-check-target-7gk2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876583 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg podName:4aa4af63-23fe-4afd-ba45-8d1926913384 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:08.876561547 +0000 UTC m=+18.006298113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqxlg" (UniqueName: "kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg") pod "network-check-target-7gk2l" (UID: "4aa4af63-23fe-4afd-ba45-8d1926913384") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876645 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:00.876708 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:00.876678 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:08.876666105 +0000 UTC m=+18.006402684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:01.309757 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:01.309719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:01.310036 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:01.310006 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:01.310506 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:01.310468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:01.310629 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:01.310606 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:01.310781 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:01.310755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:01.310867 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:01.310842 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:03.308607 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:03.308095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:03.308607 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:03.308055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:03.308607 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:03.308231 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:03.308607 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:03.308351 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:03.308607 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:03.308101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:03.308607 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:03.308458 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:05.307460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:05.307429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:05.307460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:05.307451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:05.307967 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:05.307452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:05.307967 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:05.307539 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:05.307967 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:05.307629 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:05.307967 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:05.307733 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:07.308232 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:07.308199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:07.308720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:07.308199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:07.308720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:07.308330 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:07.308720 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:07.308331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:07.308720 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:07.308425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:07.308720 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:07.308497 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:08.943317 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:08.943282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:08.943327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:08.943379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943493 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943504 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943519 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943529 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nqxlg for pod openshift-network-diagnostics/network-check-target-7gk2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943504 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943570 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:24.943551392 +0000 UTC m=+34.073287966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943588 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg podName:4aa4af63-23fe-4afd-ba45-8d1926913384 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:24.943579117 +0000 UTC m=+34.073315682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqxlg" (UniqueName: "kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg") pod "network-check-target-7gk2l" (UID: "4aa4af63-23fe-4afd-ba45-8d1926913384") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:08.943776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:08.943605 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret podName:73b7914e-e824-44f9-8a97-d735b8de33a7 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:24.943597532 +0000 UTC m=+34.073334094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret") pod "global-pull-secret-syncer-6v4bn" (UID: "73b7914e-e824-44f9-8a97-d735b8de33a7") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:09.308034 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:09.308006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:09.308185 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:09.308007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:09.308185 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:09.308118 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:09.308283 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:09.308247 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:09.308339 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:09.308305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:09.308456 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:09.308436 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:11.308151 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:11.308121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:11.308620 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:11.308180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:11.308620 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:11.308246 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:11.308620 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:11.308323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:11.308620 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:11.308463 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:11.308620 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:11.308570 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:13.307711 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.307685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:13.308367 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.307685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:13.308367 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:13.307809 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:13.308367 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:13.307881 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:13.308367 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.307685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:13.308367 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:13.308021 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:13.471926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.471866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tjzq" event={"ID":"97adad06-f0b5-4467-b20f-9dafdc3058a6","Type":"ContainerStarted","Data":"53f3955f749fa562b92d88a258f1c22997f52311a2012e25818afa5227de2141"} Sep 15 20:37:13.473372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.473348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" event={"ID":"785472ab-8c31-4ca5-bfd7-07b33eeaa1b2","Type":"ContainerStarted","Data":"e1c706aba58dc2e6ae90dc7287891bb93335dc737e36db11c7224203d2841fca"} Sep 15 20:37:13.475727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.475695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"9c068292cdc44e46872f7cefd159df3e6d6a7e88b11007d0fc53fa7eea190d59"} Sep 15 20:37:13.475798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.475761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"d7b07556d4867c396c4243e84e88a52493647a056d2a5f99abe0e2141f8da8a6"} Sep 15 20:37:13.475850 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.475805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"0ff078bd5c5bbc19ed4bcc1ccfe530a4492378304de601cbafd642f1ed04ea54"} Sep 15 20:37:13.477178 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.477150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vjcfq" event={"ID":"a11c1990-58c3-4921-84c7-f7bf86deaddf","Type":"ContainerStarted","Data":"83f622b2df2c8c9530a7fb7793c242cf1a876705498814daaacb0e536545787e"} Sep 15 20:37:13.478829 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.478801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ckvww" event={"ID":"b2d6afa5-55c1-484f-9539-b6d240fdf155","Type":"ContainerStarted","Data":"22570fc9d7398da93e58530ed8da8395b1d0bafe594effcd2e40535789d80b52"} Sep 15 20:37:13.480325 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.480299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" event={"ID":"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3","Type":"ContainerStarted","Data":"2cef30df1c45ba572a9f452cbbaedb593b1f4a4dcd169c3632796989a710e39d"} Sep 15 20:37:13.481883 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.481859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ssxn" event={"ID":"0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809","Type":"ContainerStarted","Data":"61f75a75e580181c09f90c6d5c14c74f3f7bb8f9b85663b54b6570bd4f4f66fe"} Sep 15 20:37:13.483451 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.483431 2580 generic.go:356] "Generic (PLEG): container finished" podID="43127936-57a8-463b-ac1b-2568e6fa2347" containerID="731bb0dd2f17494f107bb29fa546512c50591e7ca31c8236e0c01a6f1f3d1b41" exitCode=0 Sep 15 20:37:13.483598 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.483458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerDied","Data":"731bb0dd2f17494f107bb29fa546512c50591e7ca31c8236e0c01a6f1f3d1b41"} Sep 15 20:37:13.485308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.485259 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7tjzq" podStartSLOduration=9.891833832 podStartE2EDuration="21.485248107s" totalImagesPullingTime="11.593414275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:13.485162833 +0000 UTC m=+22.614899416" watchObservedRunningTime="2026-09-15 20:37:13.485248107 +0000 UTC m=+22.614984691" Sep 15 20:37:13.485717 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.485692 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" podStartSLOduration=20.485685295 podStartE2EDuration="20.485685295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:36:56.451576156 +0000 UTC m=+5.581312743" watchObservedRunningTime="2026-09-15 20:37:13.485685295 +0000 UTC m=+22.615421882" Sep 15 20:37:13.497643 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.497609 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vjcfq" podStartSLOduration=2.868961451 podStartE2EDuration="21.497597162s" totalImagesPullingTime="18.628635711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:13.497464163 +0000 UTC m=+22.627200745" watchObservedRunningTime="2026-09-15 20:37:13.497597162 +0000 UTC m=+22.627333745" Sep 15 20:37:13.513033 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.512992 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pr5lj" podStartSLOduration=2.880499093 podStartE2EDuration="21.512975522s" totalImagesPullingTime="18.632476429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:13.512170359 +0000 UTC m=+22.641906942" watchObservedRunningTime="2026-09-15 20:37:13.512975522 +0000 UTC m=+22.642712106" Sep 15 20:37:13.529581 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.529549 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ckvww" podStartSLOduration=2.8461049259999998 podStartE2EDuration="21.529536054s" totalImagesPullingTime="18.683431128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:13.529337732 +0000 UTC m=+22.659074316" watchObservedRunningTime="2026-09-15 20:37:13.529536054 +0000 UTC m=+22.659272637" Sep 15 20:37:13.542896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:13.542864 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2ssxn" podStartSLOduration=2.918333413 podStartE2EDuration="21.542852957s" totalImagesPullingTime="18.624519544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:13.542664735 +0000 UTC m=+22.672401318" watchObservedRunningTime="2026-09-15 20:37:13.542852957 +0000 UTC m=+22.672589540" Sep 15 20:37:14.379000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:14.378975 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 20:37:14.487140 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:14.487107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4w2w2" event={"ID":"4a1999b9-32f4-4058-aef0-078fbd9ca031","Type":"ContainerStarted","Data":"9c6da0d244a1580b72efadecaf4cbaaa626d51af569601c0a1b977b31702fded"} Sep 15 20:37:14.489009 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:14.488984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" event={"ID":"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3","Type":"ContainerStarted","Data":"a338872ea9283316085938db6c28df5afeccf4120a93e2bf000c3697102dff21"} Sep 15 20:37:14.492167 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:14.492140 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"e028d6c7708e5979b245c105e96a456ebe9e5eead800125eb53515e96a8e6734"} Sep 15 20:37:14.492264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:14.492205 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"ffcce113ead6ce5890616448a61851b818845457310169b104fde5e051610ddc"} Sep 15 20:37:14.492264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:14.492236 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"b9dca0daf2addc8a3ed5acf0230bf18ef0f5999b491e73486b2b9c8a3b035f82"} Sep 15 20:37:14.505628 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:14.505580 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4w2w2" podStartSLOduration=3.884827765 podStartE2EDuration="22.505560417s" totalImagesPullingTime="18.620732652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:14.504941013 +0000 UTC m=+23.634677598" watchObservedRunningTime="2026-09-15 20:37:14.505560417 +0000 UTC m=+23.635297005" Sep 15 20:37:15.259650 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:15.259532 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T20:37:14.378995605Z","UUID":"5b2ad95e-1b5a-4a21-99dd-94a527329cf7","Handler":null,"Name":"","Endpoint":""} Sep 15 20:37:15.261212 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:15.261149 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 20:37:15.261212 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:15.261185 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 20:37:15.307586 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:15.307558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:15.307743 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:15.307558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:15.307743 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:15.307718 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:15.307854 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:15.307768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:15.307854 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:15.307558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:15.307956 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:15.307857 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:16.501278 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:16.501243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"557c00f15101666f7a4e884cdefefa207b74d88aa1203ada6f905767d2d3d05e"} Sep 15 20:37:16.503922 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:16.503889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" event={"ID":"4f0aef44-c4ae-42a0-aa65-0abbf2c0b1b3","Type":"ContainerStarted","Data":"a33cab95b244f6a8f8d76de9975849a3b4e6d7cc301d4540ed23e5def39d79ee"} Sep 15 20:37:16.523204 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:16.523158 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wx6x" podStartSLOduration=3.7377855970000002 podStartE2EDuration="24.523139929s" totalImagesPullingTime="20.785354332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:16.519995102 +0000 UTC m=+25.649731675" watchObservedRunningTime="2026-09-15 20:37:16.523139929 +0000 UTC m=+25.652876513" Sep 15 20:37:17.026424 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:17.026381 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:37:17.027047 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:17.027023 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:37:17.308261 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:17.308183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:17.308261 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:17.308230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:17.308517 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:17.308328 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:17.308517 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:17.308391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:17.308517 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:17.308440 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:17.308626 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:17.308544 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:17.507085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:17.507039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:37:17.507532 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:17.507494 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7tjzq" Sep 15 20:37:18.511154 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:18.510896 2580 generic.go:356] "Generic (PLEG): container finished" podID="43127936-57a8-463b-ac1b-2568e6fa2347" containerID="177a9c010875dc63bcc8d10f8b9dc1102228594046b53558114898b790dd44c3" exitCode=0 Sep 15 20:37:18.511558 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:18.510972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerDied","Data":"177a9c010875dc63bcc8d10f8b9dc1102228594046b53558114898b790dd44c3"} Sep 15 20:37:18.517604 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:18.517572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" event={"ID":"0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397","Type":"ContainerStarted","Data":"56d48e623b9a58d8d180b38ecf4300d1d37d05c07841fb8f1625e2fa13ef4d9c"} Sep 15 20:37:18.552842 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:18.552804 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" podStartSLOduration=7.881975946 podStartE2EDuration="26.552790787s" totalImagesPullingTime="18.670814841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:18.551658136 +0000 UTC m=+27.681394719" watchObservedRunningTime="2026-09-15 20:37:18.552790787 +0000 UTC m=+27.682527369" Sep 15 20:37:19.311182 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.311154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:19.311318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.311268 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:19.311318 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:19.311275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:19.311318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.311306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:19.311489 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:19.311417 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:19.311557 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:19.311524 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:19.521893 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.521862 2580 generic.go:356] "Generic (PLEG): container finished" podID="43127936-57a8-463b-ac1b-2568e6fa2347" containerID="cf635093dac28368d1331ef31ee97da87de7ccbecf0f2c6e7bd8cdc1888a298c" exitCode=0 Sep 15 20:37:19.522704 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.521939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerDied","Data":"cf635093dac28368d1331ef31ee97da87de7ccbecf0f2c6e7bd8cdc1888a298c"} Sep 15 20:37:19.522704 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.522021 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 20:37:19.522978 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.522704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:37:19.522978 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.522795 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:37:19.538743 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.538719 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:37:19.539298 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.539276 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:37:19.733188 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.733148 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:19.801051 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.800974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6v4bn"] Sep 15 20:37:19.801191 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.801063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:19.801191 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:19.801147 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:19.804240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.804213 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7gk2l"] Sep 15 20:37:19.804362 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.804304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:19.804454 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:19.804396 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:19.804976 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.804954 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5vvbb"] Sep 15 20:37:19.805069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.805043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:19.805158 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:19.805140 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:19.823106 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.823080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:19.880526 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:19.880502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:20.526496 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:20.526305 2580 generic.go:356] "Generic (PLEG): container finished" podID="43127936-57a8-463b-ac1b-2568e6fa2347" containerID="56ab304b66dc302c2cb89d445dcaf5b3207ac060eabb944a314497a633b83a67" exitCode=0 Sep 15 20:37:20.526968 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:20.526388 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerDied","Data":"56ab304b66dc302c2cb89d445dcaf5b3207ac060eabb944a314497a633b83a67"} Sep 15 20:37:20.526968 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:20.526601 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 20:37:21.105495 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:21.105456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:21.311385 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:21.311351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:21.311557 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:21.311461 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:21.311557 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:21.311474 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:21.311678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:21.311561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:21.311678 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:21.311617 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:21.311678 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:21.311553 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:21.529272 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:21.529242 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 20:37:21.685389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:21.685348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:21.785973 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:21.785885 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:22.942635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:22.942311 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:37:22.943062 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:22.943031 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 20:37:22.955537 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:22.955489 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" podUID="0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397" containerName="ovnkube-controller" probeResult="failure" output="" Sep 15 20:37:22.964963 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:22.964928 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" podUID="0fad074d-4b1c-4a3e-aff3-d9a0fd4d1397" containerName="ovnkube-controller" probeResult="failure" output="" Sep 15 20:37:23.308148 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:23.308115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:23.308148 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:23.308148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:23.308359 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:23.308118 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:23.308359 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:23.308240 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:23.308359 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:23.308335 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:23.308525 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:23.308388 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:23.384024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:23.383989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:24.956873 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:24.956847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:24.956897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:24.956915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.956996 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.957006 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.957025 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.957047 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.957051 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:56.957038 +0000 UTC m=+66.086774561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.957061 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nqxlg for pod openshift-network-diagnostics/network-check-target-7gk2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.957064 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret podName:73b7914e-e824-44f9-8a97-d735b8de33a7 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:56.957058118 +0000 UTC m=+66.086794679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret") pod "global-pull-secret-syncer-6v4bn" (UID: "73b7914e-e824-44f9-8a97-d735b8de33a7") : object "kube-system"/"original-pull-secret" not registered Sep 15 20:37:24.957186 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:24.957098 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg podName:4aa4af63-23fe-4afd-ba45-8d1926913384 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:56.957086032 +0000 UTC m=+66.086822593 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nqxlg" (UniqueName: "kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg") pod "network-check-target-7gk2l" (UID: "4aa4af63-23fe-4afd-ba45-8d1926913384") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 20:37:25.307789 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.307623 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:25.308001 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:25.307850 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gk2l" podUID="4aa4af63-23fe-4afd-ba45-8d1926913384" Sep 15 20:37:25.308001 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.307703 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:25.308001 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:25.307914 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6v4bn" podUID="73b7914e-e824-44f9-8a97-d735b8de33a7" Sep 15 20:37:25.308001 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.307623 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:25.308001 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:25.307975 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:37:25.539452 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.539422 2580 generic.go:356] "Generic (PLEG): container finished" podID="43127936-57a8-463b-ac1b-2568e6fa2347" containerID="944881b4e94c4ca29c0d0b2ed2cf9f9e7e328493fc8cbd312b94d7703e4325ea" exitCode=0 Sep 15 20:37:25.539609 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.539465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerDied","Data":"944881b4e94c4ca29c0d0b2ed2cf9f9e7e328493fc8cbd312b94d7703e4325ea"} Sep 15 20:37:25.682121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.682103 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeReady" Sep 15 20:37:25.682209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.682205 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 20:37:25.721380 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.721361 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-25vtb"] Sep 15 20:37:25.735424 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.735381 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gnbcs"] Sep 15 20:37:25.735519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.735502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.738111 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.738092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 20:37:25.738210 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.738188 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 20:37:25.738311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.738296 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx25c\"" Sep 15 20:37:25.747435 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.747398 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25vtb"] Sep 15 20:37:25.747511 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.747454 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gnbcs"] Sep 15 20:37:25.747548 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.747517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:25.749272 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.749258 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 20:37:25.749786 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.749766 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4qdwd\"" Sep 15 20:37:25.749933 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.749800 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 20:37:25.752435 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.750648 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 20:37:25.862853 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.862799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.862853 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.862830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-tmp-dir\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.862853 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.862849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-config-volume\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.863021 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.862900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2nq\" (UniqueName: \"kubernetes.io/projected/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-kube-api-access-4f2nq\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.863021 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.862921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869hv\" (UniqueName: \"kubernetes.io/projected/bd61f39f-8693-4da4-b81a-b84b3ade845f-kube-api-access-869hv\") pod \"ingress-canary-gnbcs\" (UID: \"bd61f39f-8693-4da4-b81a-b84b3ade845f\") " pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:25.863021 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.862943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd61f39f-8693-4da4-b81a-b84b3ade845f-cert\") pod \"ingress-canary-gnbcs\" (UID: \"bd61f39f-8693-4da4-b81a-b84b3ade845f\") " pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:25.964081 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.964427 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-tmp-dir\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.964427 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-config-volume\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.964427 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2nq\" (UniqueName: \"kubernetes.io/projected/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-kube-api-access-4f2nq\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.964427 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869hv\" (UniqueName: \"kubernetes.io/projected/bd61f39f-8693-4da4-b81a-b84b3ade845f-kube-api-access-869hv\") pod \"ingress-canary-gnbcs\" (UID: \"bd61f39f-8693-4da4-b81a-b84b3ade845f\") " pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:25.964427 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:25.964195 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:25.964427 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd61f39f-8693-4da4-b81a-b84b3ade845f-cert\") pod \"ingress-canary-gnbcs\" (UID: \"bd61f39f-8693-4da4-b81a-b84b3ade845f\") " pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:25.964427 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:25.964286 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:37:26.464268533 +0000 UTC m=+35.594005095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:37:25.964672 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-tmp-dir\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.964672 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.964611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-config-volume\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.968355 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.968322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd61f39f-8693-4da4-b81a-b84b3ade845f-cert\") pod \"ingress-canary-gnbcs\" (UID: \"bd61f39f-8693-4da4-b81a-b84b3ade845f\") " pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:25.971952 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.971920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2nq\" (UniqueName: \"kubernetes.io/projected/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-kube-api-access-4f2nq\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:25.972154 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:25.972134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-869hv\" (UniqueName: \"kubernetes.io/projected/bd61f39f-8693-4da4-b81a-b84b3ade845f-kube-api-access-869hv\") pod \"ingress-canary-gnbcs\" (UID: \"bd61f39f-8693-4da4-b81a-b84b3ade845f\") " pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:26.105448 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:26.105398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gnbcs" Sep 15 20:37:26.427772 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:26.427701 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gnbcs"] Sep 15 20:37:26.432233 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:37:26.432205 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd61f39f_8693_4da4_b81a_b84b3ade845f.slice/crio-300718fef683678d2b91e7f03725fb109f4a7abd27c4c494bb018275e07fab3f WatchSource:0}: Error finding container 300718fef683678d2b91e7f03725fb109f4a7abd27c4c494bb018275e07fab3f: Status 404 returned error can't find the container with id 300718fef683678d2b91e7f03725fb109f4a7abd27c4c494bb018275e07fab3f Sep 15 20:37:26.469617 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:26.469594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:26.469722 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:26.469709 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:26.469764 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:26.469757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:37:27.469744207 +0000 UTC m=+36.599480768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:37:26.543221 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:26.543192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gnbcs" event={"ID":"bd61f39f-8693-4da4-b81a-b84b3ade845f","Type":"ContainerStarted","Data":"300718fef683678d2b91e7f03725fb109f4a7abd27c4c494bb018275e07fab3f"} Sep 15 20:37:26.546359 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:26.546329 2580 generic.go:356] "Generic (PLEG): container finished" podID="43127936-57a8-463b-ac1b-2568e6fa2347" containerID="59b7b6986968897b04d3eaeaa765e56d63064b32bd0ded0d7fef61fdd5cbdd4c" exitCode=0 Sep 15 20:37:26.546489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:26.546366 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerDied","Data":"59b7b6986968897b04d3eaeaa765e56d63064b32bd0ded0d7fef61fdd5cbdd4c"} Sep 15 20:37:26.998087 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:26.997859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:27.057778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.057747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:27.307595 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.307564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:27.307766 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.307564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:27.307766 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.307564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:27.310354 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.310332 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 20:37:27.311025 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.311007 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 20:37:27.311218 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.311196 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 20:37:27.311293 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.311244 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxplx\"" Sep 15 20:37:27.311491 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.311476 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 20:37:27.311557 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.311526 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzk8x\"" Sep 15 20:37:27.477632 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.477599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:27.477822 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:27.477753 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:27.477894 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:27.477830 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:37:29.477805726 +0000 UTC m=+38.607542289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:37:27.552901 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.552858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" event={"ID":"43127936-57a8-463b-ac1b-2568e6fa2347","Type":"ContainerStarted","Data":"7b38dd1ae5180e7814cee243defd1371c3c05ea8e0472c0de007fdd626b1d3f9"} Sep 15 20:37:27.572906 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.572823 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q4jtq" podStartSLOduration=7.157425347 podStartE2EDuration="35.572809402s" totalImagesPullingTime="28.415384055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:27.57215092 +0000 UTC m=+36.701887513" watchObservedRunningTime="2026-09-15 20:37:27.572809402 +0000 UTC m=+36.702546008" Sep 15 20:37:27.692058 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.692022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:27.794152 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:27.794116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:29.118372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:29.118341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:29.172965 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:29.172940 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:29.496209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:29.496181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:29.496302 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:29.496293 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:29.496356 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:29.496347 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:37:33.496333727 +0000 UTC m=+42.626070288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:37:29.559510 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:29.559480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gnbcs" event={"ID":"bd61f39f-8693-4da4-b81a-b84b3ade845f","Type":"ContainerStarted","Data":"bedc1dae98e647c7af92ce4712b7fdcb063e530e4d19b2c82449f4636f52cac2"} Sep 15 20:37:29.574228 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:29.574183 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gnbcs" podStartSLOduration=1.909504155 podStartE2EDuration="4.5741683s" totalImagesPullingTime="2.664664145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:29.573161006 +0000 UTC m=+38.702897589" watchObservedRunningTime="2026-09-15 20:37:29.5741683 +0000 UTC m=+38.703904879" Sep 15 20:37:31.215738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:31.215701 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:31.268673 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:31.268644 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:33.524301 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:33.524263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:33.524652 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:33.524387 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:33.524652 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:33.524467 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:37:41.524452057 +0000 UTC m=+50.654188617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:37:41.577392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:41.577356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:41.577827 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:41.577508 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:41.577827 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:41.577582 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:37:57.577565492 +0000 UTC m=+66.707302053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:37:42.696693 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:42.696663 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:42.762756 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:42.762728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:48.986989 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:48.986953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:49.039130 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:49.039108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:52.965370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:52.965339 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzkb9" Sep 15 20:37:53.077096 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:53.077063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:53.134386 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:53.134358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:53.215246 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:53.215217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:56.971918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.971871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:56.972309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.971932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:56.972309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.971954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:37:56.979122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.979095 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 20:37:56.979250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.979106 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 20:37:56.979250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.979161 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 20:37:56.983037 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:56.983020 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 15 20:37:56.983103 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:56.983093 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:39:00.983072111 +0000 UTC m=+130.112808671 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : secret "metrics-daemon-secret" not found Sep 15 20:37:56.984581 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.984563 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 20:37:56.986004 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.985978 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73b7914e-e824-44f9-8a97-d735b8de33a7-original-pull-secret\") pod \"global-pull-secret-syncer-6v4bn\" (UID: \"73b7914e-e824-44f9-8a97-d735b8de33a7\") " pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:56.996005 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:56.995986 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxlg\" (UniqueName: \"kubernetes.io/projected/4aa4af63-23fe-4afd-ba45-8d1926913384-kube-api-access-nqxlg\") pod \"network-check-target-7gk2l\" (UID: \"4aa4af63-23fe-4afd-ba45-8d1926913384\") " pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:57.019089 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.019070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6v4bn" Sep 15 20:37:57.033773 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.033755 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzk8x\"" Sep 15 20:37:57.042274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.042254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:37:57.286001 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.285972 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6v4bn"] Sep 15 20:37:57.289192 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:37:57.289168 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b7914e_e824_44f9_8a97_d735b8de33a7.slice/crio-3d50ee6374abf43db4d2afac93ae63a923291b634d63322a22a8f6ecd8591835 WatchSource:0}: Error finding container 3d50ee6374abf43db4d2afac93ae63a923291b634d63322a22a8f6ecd8591835: Status 404 returned error can't find the container with id 3d50ee6374abf43db4d2afac93ae63a923291b634d63322a22a8f6ecd8591835 Sep 15 20:37:57.308019 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.307994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7gk2l"] Sep 15 20:37:57.313470 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:37:57.313400 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa4af63_23fe_4afd_ba45_8d1926913384.slice/crio-47c604096acfcb3aa9c8e8c3ee68fbfbdcd3d3b462de33bea1babfe2971fcc0b WatchSource:0}: Error finding container 47c604096acfcb3aa9c8e8c3ee68fbfbdcd3d3b462de33bea1babfe2971fcc0b: Status 404 returned error can't find the container with id 47c604096acfcb3aa9c8e8c3ee68fbfbdcd3d3b462de33bea1babfe2971fcc0b Sep 15 20:37:57.627175 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.627096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7gk2l" event={"ID":"4aa4af63-23fe-4afd-ba45-8d1926913384","Type":"ContainerStarted","Data":"47c604096acfcb3aa9c8e8c3ee68fbfbdcd3d3b462de33bea1babfe2971fcc0b"} Sep 15 20:37:57.628299 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.628268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6v4bn" event={"ID":"73b7914e-e824-44f9-8a97-d735b8de33a7","Type":"ContainerStarted","Data":"3d50ee6374abf43db4d2afac93ae63a923291b634d63322a22a8f6ecd8591835"} Sep 15 20:37:57.676477 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:57.676435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:37:57.676630 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:57.676606 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:37:57.676694 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:37:57.676682 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:38:29.676661801 +0000 UTC m=+98.806398364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:37:58.374523 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:58.374483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:37:58.430146 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:37:58.430108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:38:01.610026 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.609994 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp"] Sep 15 20:38:01.614004 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.613955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:01.616486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.616458 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 15 20:38:01.617604 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.617580 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 15 20:38:01.617875 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.617859 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qjg47\"" Sep 15 20:38:01.618078 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.618062 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 15 20:38:01.618316 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.618291 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 15 20:38:01.621182 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.621036 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp"] Sep 15 20:38:01.640517 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.640492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7gk2l" event={"ID":"4aa4af63-23fe-4afd-ba45-8d1926913384","Type":"ContainerStarted","Data":"82cb46cda3604023cf62f4be7ac310264b2c15a3cbd937413d4ca42c45a3e42e"} Sep 15 20:38:01.640620 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.640612 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:38:01.656687 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.656652 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7gk2l" podStartSLOduration=65.402859515 podStartE2EDuration="1m9.656639396s" totalImagesPullingTime="4.253779881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:38:01.656476826 +0000 UTC m=+70.786213411" watchObservedRunningTime="2026-09-15 20:38:01.656639396 +0000 UTC m=+70.786375979" Sep 15 20:38:01.707988 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.707949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ee79ab4-9232-4570-a689-9fb061c6e013-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7955457f64-zhprp\" (UID: \"0ee79ab4-9232-4570-a689-9fb061c6e013\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:01.708124 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.707994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsl2\" (UniqueName: \"kubernetes.io/projected/0ee79ab4-9232-4570-a689-9fb061c6e013-kube-api-access-2hsl2\") pod \"managed-serviceaccount-addon-agent-7955457f64-zhprp\" (UID: \"0ee79ab4-9232-4570-a689-9fb061c6e013\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:01.809250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.809214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ee79ab4-9232-4570-a689-9fb061c6e013-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7955457f64-zhprp\" (UID: \"0ee79ab4-9232-4570-a689-9fb061c6e013\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:01.809378 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.809256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsl2\" (UniqueName: \"kubernetes.io/projected/0ee79ab4-9232-4570-a689-9fb061c6e013-kube-api-access-2hsl2\") pod \"managed-serviceaccount-addon-agent-7955457f64-zhprp\" (UID: \"0ee79ab4-9232-4570-a689-9fb061c6e013\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:01.811753 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.811722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ee79ab4-9232-4570-a689-9fb061c6e013-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7955457f64-zhprp\" (UID: \"0ee79ab4-9232-4570-a689-9fb061c6e013\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:01.817541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.817488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsl2\" (UniqueName: \"kubernetes.io/projected/0ee79ab4-9232-4570-a689-9fb061c6e013-kube-api-access-2hsl2\") pod \"managed-serviceaccount-addon-agent-7955457f64-zhprp\" (UID: \"0ee79ab4-9232-4570-a689-9fb061c6e013\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:01.933178 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:01.933142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" Sep 15 20:38:02.196357 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:02.196285 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp"] Sep 15 20:38:02.199613 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:38:02.199588 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ee79ab4_9232_4570_a689_9fb061c6e013.slice/crio-6b4d132458c79d57689ebae9b00eb3a63a1195cc9c976ccf9706feaf1db513b4 WatchSource:0}: Error finding container 6b4d132458c79d57689ebae9b00eb3a63a1195cc9c976ccf9706feaf1db513b4: Status 404 returned error can't find the container with id 6b4d132458c79d57689ebae9b00eb3a63a1195cc9c976ccf9706feaf1db513b4 Sep 15 20:38:02.644495 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:02.644458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6v4bn" event={"ID":"73b7914e-e824-44f9-8a97-d735b8de33a7","Type":"ContainerStarted","Data":"b09828e3240e963c0ea9ab0a51b1a20f5dc86107e46517ce6731ff05606e7a1c"} Sep 15 20:38:02.645750 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:02.645729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" event={"ID":"0ee79ab4-9232-4570-a689-9fb061c6e013","Type":"ContainerStarted","Data":"6b4d132458c79d57689ebae9b00eb3a63a1195cc9c976ccf9706feaf1db513b4"} Sep 15 20:38:02.658230 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:02.658186 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6v4bn" podStartSLOduration=66.376378843 podStartE2EDuration="1m10.658171733s" totalImagesPullingTime="4.28179289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:38:02.657972408 +0000 UTC m=+71.787708991" watchObservedRunningTime="2026-09-15 20:38:02.658171733 +0000 UTC m=+71.787908316" Sep 15 20:38:05.656080 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:05.656044 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" event={"ID":"0ee79ab4-9232-4570-a689-9fb061c6e013","Type":"ContainerStarted","Data":"d5ec031459a8b50608d7b0c30f7bb5141549219af6fbc55a804fd25ddd57a135"} Sep 15 20:38:05.674061 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:05.674009 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" podStartSLOduration=2.037907493 podStartE2EDuration="4.673995458s" totalImagesPullingTime="2.636087965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:38:05.672366029 +0000 UTC m=+74.802102613" watchObservedRunningTime="2026-09-15 20:38:05.673995458 +0000 UTC m=+74.803732040" Sep 15 20:38:22.002951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:22.002814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:38:22.052381 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:22.052343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:38:29.682349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:29.682268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:38:29.682714 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:38:29.682380 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:38:29.682714 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:38:29.682457 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:39:33.682441731 +0000 UTC m=+162.812178292 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:38:32.648668 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:38:32.648636 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7gk2l" Sep 15 20:39:00.990166 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:39:00.990128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:39:00.990655 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:39:00.990251 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 15 20:39:00.990655 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:39:00.990312 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:02.99029734 +0000 UTC m=+252.120033900 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : secret "metrics-daemon-secret" not found Sep 15 20:39:28.800097 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:39:28.800049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-25vtb" podUID="8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b" Sep 15 20:39:28.852057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:39:28.852028 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25vtb" Sep 15 20:39:30.326377 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:39:30.326339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:39:33.699625 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:39:33.699575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:39:33.700016 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:39:33.699696 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 20:39:33.700016 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:39:33.699749 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls podName:8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b nodeName:}" failed. No retries permitted until 2026-09-15 20:41:35.699735502 +0000 UTC m=+284.829472063 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls") pod "dns-default-25vtb" (UID: "8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b") : secret "dns-default-metrics-tls" not found Sep 15 20:39:45.307656 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:39:45.307572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:40:05.941230 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:40:05.941194 2580 generic.go:356] "Generic (PLEG): container finished" podID="0ee79ab4-9232-4570-a689-9fb061c6e013" containerID="d5ec031459a8b50608d7b0c30f7bb5141549219af6fbc55a804fd25ddd57a135" exitCode=255 Sep 15 20:40:05.941230 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:40:05.941235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" event={"ID":"0ee79ab4-9232-4570-a689-9fb061c6e013","Type":"ContainerDied","Data":"d5ec031459a8b50608d7b0c30f7bb5141549219af6fbc55a804fd25ddd57a135"} Sep 15 20:40:05.946813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:40:05.946791 2580 scope.go:117] "RemoveContainer" containerID="d5ec031459a8b50608d7b0c30f7bb5141549219af6fbc55a804fd25ddd57a135" Sep 15 20:40:06.944967 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:40:06.944933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7955457f64-zhprp" event={"ID":"0ee79ab4-9232-4570-a689-9fb061c6e013","Type":"ContainerStarted","Data":"638d99cba1478903bd4aa7a4b8e015427e16cee8869edaabaceb37f18eeda544"} Sep 15 20:40:26.714423 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:40:26.714379 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:00.691863 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.691826 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6"] Sep 15 20:41:00.694670 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.694643 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8zzs5"] Sep 15 20:41:00.694837 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.694811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" Sep 15 20:41:00.697121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.697100 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dwtsg\"" Sep 15 20:41:00.697456 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.697437 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29"] Sep 15 20:41:00.697568 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.697532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.699385 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.699359 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.699519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.699500 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.699589 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.699570 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zf52m\"" Sep 15 20:41:00.699589 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.699577 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 15 20:41:00.699787 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.699773 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 15 20:41:00.700125 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.700093 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-27q29"] Sep 15 20:41:00.700254 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.700197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:00.702103 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.702080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.702486 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.702450 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 15 20:41:00.703104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.703044 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.703355 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.703334 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59f448b5b8-4bm9g"] Sep 15 20:41:00.703504 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.703458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.703583 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.703503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lxxtj\"" Sep 15 20:41:00.706102 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.705639 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.706102 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.705900 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.706102 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.706041 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kn52s\"" Sep 15 20:41:00.706809 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.706785 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 15 20:41:00.706932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.706913 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 15 20:41:00.707369 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.707353 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 15 20:41:00.707452 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.707435 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6"] Sep 15 20:41:00.707496 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.707477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8zzs5"] Sep 15 20:41:00.707780 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.707765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.709487 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.709469 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29"] Sep 15 20:41:00.713698 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.713552 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 15 20:41:00.716269 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.715595 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 15 20:41:00.716269 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.715804 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 15 20:41:00.716269 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.715822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-27q29"] Sep 15 20:41:00.716269 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.716015 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.716528 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.716307 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 15 20:41:00.716641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.716619 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jxmrj\"" Sep 15 20:41:00.717074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.716907 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.718314 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.718295 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 15 20:41:00.723094 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.723074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59f448b5b8-4bm9g"] Sep 15 20:41:00.761738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761710 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwgv\" (UniqueName: \"kubernetes.io/projected/9312d9bc-0d1b-48c8-8e90-ebe08397042f-kube-api-access-hjwgv\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.761849 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-default-certificate\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.761849 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs69h\" (UniqueName: \"kubernetes.io/projected/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-kube-api-access-xs69h\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.761849 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/057261c3-367a-40b0-ba97-328f74076ee0-serving-cert\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.761849 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/057261c3-367a-40b0-ba97-328f74076ee0-trusted-ca\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.762039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9312d9bc-0d1b-48c8-8e90-ebe08397042f-service-ca-bundle\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.762039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r85jz\" (UniqueName: \"kubernetes.io/projected/f7efe8ad-7aed-4e2e-a53a-4e233a258eeb-kube-api-access-r85jz\") pod \"network-check-source-6f8b768455-2rjq6\" (UID: \"f7efe8ad-7aed-4e2e-a53a-4e233a258eeb\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" Sep 15 20:41:00.762039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.761996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.762039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9312d9bc-0d1b-48c8-8e90-ebe08397042f-snapshots\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9312d9bc-0d1b-48c8-8e90-ebe08397042f-serving-cert\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpk5j\" (UniqueName: \"kubernetes.io/projected/b4a56ae9-d999-4b36-9067-3e93bd4d4958-kube-api-access-hpk5j\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-stats-auth\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vf4\" (UniqueName: \"kubernetes.io/projected/057261c3-367a-40b0-ba97-328f74076ee0-kube-api-access-s6vf4\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9312d9bc-0d1b-48c8-8e90-ebe08397042f-tmp\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.762197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/057261c3-367a-40b0-ba97-328f74076ee0-config\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.762452 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.762213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9312d9bc-0d1b-48c8-8e90-ebe08397042f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.790173 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.790145 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb"] Sep 15 20:41:00.793189 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.793170 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5846b776d8-7vlwp"] Sep 15 20:41:00.793293 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.793281 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" Sep 15 20:41:00.795803 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.795786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.796219 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.796195 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xvx95\"" Sep 15 20:41:00.796219 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.796209 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 15 20:41:00.797907 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.797891 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 20:41:00.798104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.798089 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgtm7\"" Sep 15 20:41:00.798452 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.798394 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 20:41:00.798554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.798397 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:41:00.798554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.798421 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 20:41:00.803110 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.803083 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb"] Sep 15 20:41:00.804144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.804119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 20:41:00.806880 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.806858 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5846b776d8-7vlwp"] Sep 15 20:41:00.863503 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/057261c3-367a-40b0-ba97-328f74076ee0-config\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.863629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9312d9bc-0d1b-48c8-8e90-ebe08397042f-service-ca-bundle\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.863629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/057261c3-367a-40b0-ba97-328f74076ee0-serving-cert\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.863629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.863629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9312d9bc-0d1b-48c8-8e90-ebe08397042f-snapshots\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.863629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk5j\" (UniqueName: \"kubernetes.io/projected/b4a56ae9-d999-4b36-9067-3e93bd4d4958-kube-api-access-hpk5j\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-stats-auth\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9312d9bc-0d1b-48c8-8e90-ebe08397042f-tmp\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vf4\" (UniqueName: \"kubernetes.io/projected/057261c3-367a-40b0-ba97-328f74076ee0-kube-api-access-s6vf4\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-certificates\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.863752 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-bound-sa-token\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.863830 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:01.363807337 +0000 UTC m=+250.493543912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : secret "router-metrics-certs-default" not found Sep 15 20:41:00.863867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzc69\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-kube-api-access-kzc69\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.863893 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.863951 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls podName:b4a56ae9-d999-4b36-9067-3e93bd4d4958 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:01.363933128 +0000 UTC m=+250.493669688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dgq29" (UID: "b4a56ae9-d999-4b36-9067-3e93bd4d4958") : secret "samples-operator-tls" not found Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs69h\" (UniqueName: \"kubernetes.io/projected/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-kube-api-access-xs69h\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.863979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/057261c3-367a-40b0-ba97-328f74076ee0-trusted-ca\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-image-registry-private-configuration\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r85jz\" (UniqueName: \"kubernetes.io/projected/f7efe8ad-7aed-4e2e-a53a-4e233a258eeb-kube-api-access-r85jz\") pod \"network-check-source-6f8b768455-2rjq6\" (UID: \"f7efe8ad-7aed-4e2e-a53a-4e233a258eeb\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchc2\" (UniqueName: \"kubernetes.io/projected/1e360961-0f5b-42b6-8a9b-576067c2e536-kube-api-access-fchc2\") pod \"volume-data-source-validator-86695b7cc7-2c6tb\" (UID: \"1e360961-0f5b-42b6-8a9b-576067c2e536\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9312d9bc-0d1b-48c8-8e90-ebe08397042f-service-ca-bundle\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9312d9bc-0d1b-48c8-8e90-ebe08397042f-snapshots\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9312d9bc-0d1b-48c8-8e90-ebe08397042f-tmp\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.864341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/057261c3-367a-40b0-ba97-328f74076ee0-config\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-trusted-ca\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9312d9bc-0d1b-48c8-8e90-ebe08397042f-serving-cert\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-installation-pull-secrets\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9312d9bc-0d1b-48c8-8e90-ebe08397042f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.864625 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:01.364613628 +0000 UTC m=+250.494350190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : configmap references non-existent config key: service-ca.crt Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ffc2beb-5319-45f1-851b-eafab7fd67a5-ca-trust-extracted\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwgv\" (UniqueName: \"kubernetes.io/projected/9312d9bc-0d1b-48c8-8e90-ebe08397042f-kube-api-access-hjwgv\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.865029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.864738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-default-certificate\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.865635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.865615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/057261c3-367a-40b0-ba97-328f74076ee0-trusted-ca\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.865825 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.865804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9312d9bc-0d1b-48c8-8e90-ebe08397042f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.866762 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.866740 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/057261c3-367a-40b0-ba97-328f74076ee0-serving-cert\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.866882 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.866761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-stats-auth\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.866950 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.866930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9312d9bc-0d1b-48c8-8e90-ebe08397042f-serving-cert\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.867038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.867018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-default-certificate\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.877330 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.877304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs69h\" (UniqueName: \"kubernetes.io/projected/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-kube-api-access-xs69h\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:00.877512 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.877488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vf4\" (UniqueName: \"kubernetes.io/projected/057261c3-367a-40b0-ba97-328f74076ee0-kube-api-access-s6vf4\") pod \"console-operator-696884fb9c-8zzs5\" (UID: \"057261c3-367a-40b0-ba97-328f74076ee0\") " pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:00.877626 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.877604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r85jz\" (UniqueName: \"kubernetes.io/projected/f7efe8ad-7aed-4e2e-a53a-4e233a258eeb-kube-api-access-r85jz\") pod \"network-check-source-6f8b768455-2rjq6\" (UID: \"f7efe8ad-7aed-4e2e-a53a-4e233a258eeb\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" Sep 15 20:41:00.877695 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.877641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk5j\" (UniqueName: \"kubernetes.io/projected/b4a56ae9-d999-4b36-9067-3e93bd4d4958-kube-api-access-hpk5j\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:00.877835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.877816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwgv\" (UniqueName: \"kubernetes.io/projected/9312d9bc-0d1b-48c8-8e90-ebe08397042f-kube-api-access-hjwgv\") pod \"insights-operator-5844b856db-27q29\" (UID: \"9312d9bc-0d1b-48c8-8e90-ebe08397042f\") " pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:00.932362 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.932324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:00.965171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-installation-pull-secrets\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ffc2beb-5319-45f1-851b-eafab7fd67a5-ca-trust-extracted\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965310 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-certificates\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965310 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-bound-sa-token\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzc69\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-kube-api-access-kzc69\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-image-registry-private-configuration\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965590 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchc2\" (UniqueName: \"kubernetes.io/projected/1e360961-0f5b-42b6-8a9b-576067c2e536-kube-api-access-fchc2\") pod \"volume-data-source-validator-86695b7cc7-2c6tb\" (UID: \"1e360961-0f5b-42b6-8a9b-576067c2e536\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" Sep 15 20:41:00.965590 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.965460 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:41:00.965590 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-trusted-ca\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965590 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.965476 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846b776d8-7vlwp: secret "image-registry-tls" not found Sep 15 20:41:00.965590 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:00.965538 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls podName:9ffc2beb-5319-45f1-851b-eafab7fd67a5 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:01.465519674 +0000 UTC m=+250.595256234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls") pod "image-registry-5846b776d8-7vlwp" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5") : secret "image-registry-tls" not found Sep 15 20:41:00.965918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ffc2beb-5319-45f1-851b-eafab7fd67a5-ca-trust-extracted\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.965918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.965898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-certificates\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.966299 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.966277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-trusted-ca\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.967593 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.967573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-installation-pull-secrets\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.967824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.967801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-image-registry-private-configuration\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.975685 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.975655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzc69\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-kube-api-access-kzc69\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:00.975685 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.975679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchc2\" (UniqueName: \"kubernetes.io/projected/1e360961-0f5b-42b6-8a9b-576067c2e536-kube-api-access-fchc2\") pod \"volume-data-source-validator-86695b7cc7-2c6tb\" (UID: \"1e360961-0f5b-42b6-8a9b-576067c2e536\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" Sep 15 20:41:00.976065 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:00.976049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-bound-sa-token\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:01.006475 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.006456 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" Sep 15 20:41:01.017498 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.017482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:01.033683 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.033658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:01.034387 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.034367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-27q29" Sep 15 20:41:01.104808 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.104757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" Sep 15 20:41:01.272237 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.272202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6"] Sep 15 20:41:01.275811 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:01.275783 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7efe8ad_7aed_4e2e_a53a_4e233a258eeb.slice/crio-738fd0d2dc5a788cd79527a8b3521540a49c062ffe0b313da40f021354d3f237 WatchSource:0}: Error finding container 738fd0d2dc5a788cd79527a8b3521540a49c062ffe0b313da40f021354d3f237: Status 404 returned error can't find the container with id 738fd0d2dc5a788cd79527a8b3521540a49c062ffe0b313da40f021354d3f237 Sep 15 20:41:01.296045 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.296022 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8zzs5"] Sep 15 20:41:01.305370 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:01.305345 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057261c3_367a_40b0_ba97_328f74076ee0.slice/crio-32339ef6ecf4452db3a313d997cae0ff17ff9c05009e11e0ac8cc0f46ec90728 WatchSource:0}: Error finding container 32339ef6ecf4452db3a313d997cae0ff17ff9c05009e11e0ac8cc0f46ec90728: Status 404 returned error can't find the container with id 32339ef6ecf4452db3a313d997cae0ff17ff9c05009e11e0ac8cc0f46ec90728 Sep 15 20:41:01.315579 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.315558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-27q29"] Sep 15 20:41:01.324310 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:01.324282 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9312d9bc_0d1b_48c8_8e90_ebe08397042f.slice/crio-43bc8ffed4ebcea9f8503df16e444f28936ad4ad17875d26d8ac5079663e9f3a WatchSource:0}: Error finding container 43bc8ffed4ebcea9f8503df16e444f28936ad4ad17875d26d8ac5079663e9f3a: Status 404 returned error can't find the container with id 43bc8ffed4ebcea9f8503df16e444f28936ad4ad17875d26d8ac5079663e9f3a Sep 15 20:41:01.368778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.368519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:01.368778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.368574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:01.368778 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.368685 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 20:41:01.368778 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.368753 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:02.368733443 +0000 UTC m=+251.498470017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : secret "router-metrics-certs-default" not found Sep 15 20:41:01.369112 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.368800 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:02.3687843 +0000 UTC m=+251.498520867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : configmap references non-existent config key: service-ca.crt Sep 15 20:41:01.369112 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.368688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:01.369112 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.368815 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 20:41:01.369112 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.368893 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls podName:b4a56ae9-d999-4b36-9067-3e93bd4d4958 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:02.368882842 +0000 UTC m=+251.498619406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dgq29" (UID: "b4a56ae9-d999-4b36-9067-3e93bd4d4958") : secret "samples-operator-tls" not found Sep 15 20:41:01.369614 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.369594 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb"] Sep 15 20:41:01.372955 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:01.372932 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e360961_0f5b_42b6_8a9b_576067c2e536.slice/crio-43339fed5607d0674def615b916a92d5dc8b9e8469fbc66ff9381eed31c4894d WatchSource:0}: Error finding container 43339fed5607d0674def615b916a92d5dc8b9e8469fbc66ff9381eed31c4894d: Status 404 returned error can't find the container with id 43339fed5607d0674def615b916a92d5dc8b9e8469fbc66ff9381eed31c4894d Sep 15 20:41:01.469318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:01.469297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:01.469483 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.469465 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:41:01.469556 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.469486 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846b776d8-7vlwp: secret "image-registry-tls" not found Sep 15 20:41:01.469556 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:01.469546 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls podName:9ffc2beb-5319-45f1-851b-eafab7fd67a5 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:02.469525726 +0000 UTC m=+251.599262306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls") pod "image-registry-5846b776d8-7vlwp" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5") : secret "image-registry-tls" not found Sep 15 20:41:02.083300 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.083265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" event={"ID":"1e360961-0f5b-42b6-8a9b-576067c2e536","Type":"ContainerStarted","Data":"43339fed5607d0674def615b916a92d5dc8b9e8469fbc66ff9381eed31c4894d"} Sep 15 20:41:02.084896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.084864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" event={"ID":"f7efe8ad-7aed-4e2e-a53a-4e233a258eeb","Type":"ContainerStarted","Data":"f32d6ced044db98b850789968861d34628a4a80d2f3c5d38ed1f6293a529a2c5"} Sep 15 20:41:02.085039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.084914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" event={"ID":"f7efe8ad-7aed-4e2e-a53a-4e233a258eeb","Type":"ContainerStarted","Data":"738fd0d2dc5a788cd79527a8b3521540a49c062ffe0b313da40f021354d3f237"} Sep 15 20:41:02.085926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.085901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-27q29" event={"ID":"9312d9bc-0d1b-48c8-8e90-ebe08397042f","Type":"ContainerStarted","Data":"43bc8ffed4ebcea9f8503df16e444f28936ad4ad17875d26d8ac5079663e9f3a"} Sep 15 20:41:02.086898 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.086877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" event={"ID":"057261c3-367a-40b0-ba97-328f74076ee0","Type":"ContainerStarted","Data":"32339ef6ecf4452db3a313d997cae0ff17ff9c05009e11e0ac8cc0f46ec90728"} Sep 15 20:41:02.099235 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.099190 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-2rjq6" podStartSLOduration=2.099176065 podStartE2EDuration="2.099176065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:02.098457535 +0000 UTC m=+251.228194118" watchObservedRunningTime="2026-09-15 20:41:02.099176065 +0000 UTC m=+251.228912647" Sep 15 20:41:02.376708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.376627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:02.376708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.376700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:02.376916 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.376736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:02.376916 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.376810 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:04.376789831 +0000 UTC m=+253.506526397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : configmap references non-existent config key: service-ca.crt Sep 15 20:41:02.376916 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.376865 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 20:41:02.376916 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.376876 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 20:41:02.377104 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.376923 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls podName:b4a56ae9-d999-4b36-9067-3e93bd4d4958 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:04.376911808 +0000 UTC m=+253.506648369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dgq29" (UID: "b4a56ae9-d999-4b36-9067-3e93bd4d4958") : secret "samples-operator-tls" not found Sep 15 20:41:02.377104 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.376937 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:04.376931409 +0000 UTC m=+253.506667969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : secret "router-metrics-certs-default" not found Sep 15 20:41:02.477440 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:02.477392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:02.477622 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.477536 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:41:02.477622 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.477556 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846b776d8-7vlwp: secret "image-registry-tls" not found Sep 15 20:41:02.477744 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:02.477625 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls podName:9ffc2beb-5319-45f1-851b-eafab7fd67a5 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:04.477598008 +0000 UTC m=+253.607334574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls") pod "image-registry-5846b776d8-7vlwp" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5") : secret "image-registry-tls" not found Sep 15 20:41:03.082608 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:03.082568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:41:03.082775 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:03.082747 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 15 20:41:03.082850 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:03.082825 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs podName:dad0237e-3438-4e30-a3f8-5f76d25d0763 nodeName:}" failed. No retries permitted until 2026-09-15 20:43:05.082807134 +0000 UTC m=+374.212543717 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs") pod "network-metrics-daemon-5vvbb" (UID: "dad0237e-3438-4e30-a3f8-5f76d25d0763") : secret "metrics-daemon-secret" not found Sep 15 20:41:04.394776 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:04.394710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:04.394776 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:04.394750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:04.395158 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:04.394798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:04.395158 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.394876 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 20:41:04.395158 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.394901 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 20:41:04.395158 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.394921 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.394907366 +0000 UTC m=+257.524643927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : configmap references non-existent config key: service-ca.crt Sep 15 20:41:04.395158 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.394945 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.394927344 +0000 UTC m=+257.524663925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : secret "router-metrics-certs-default" not found Sep 15 20:41:04.395158 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.394965 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls podName:b4a56ae9-d999-4b36-9067-3e93bd4d4958 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.394954812 +0000 UTC m=+257.524691380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dgq29" (UID: "b4a56ae9-d999-4b36-9067-3e93bd4d4958") : secret "samples-operator-tls" not found Sep 15 20:41:04.496015 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:04.495991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:04.496182 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.496160 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:41:04.496222 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.496187 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846b776d8-7vlwp: secret "image-registry-tls" not found Sep 15 20:41:04.496259 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:04.496247 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls podName:9ffc2beb-5319-45f1-851b-eafab7fd67a5 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.496228779 +0000 UTC m=+257.625965346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls") pod "image-registry-5846b776d8-7vlwp" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5") : secret "image-registry-tls" not found Sep 15 20:41:05.104391 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.104347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-27q29" event={"ID":"9312d9bc-0d1b-48c8-8e90-ebe08397042f","Type":"ContainerStarted","Data":"1dd8b47922363a87c80ee70d89572fa0c43d05d000817c5a478aa67b69cad61e"} Sep 15 20:41:05.106497 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.106467 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/0.log" Sep 15 20:41:05.106639 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.106518 2580 generic.go:356] "Generic (PLEG): container finished" podID="057261c3-367a-40b0-ba97-328f74076ee0" containerID="343b1eda449b83ca199b49dece8b661973fab800dcef30ff079f3a58b818a10c" exitCode=255 Sep 15 20:41:05.106905 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.106882 2580 scope.go:117] "RemoveContainer" containerID="343b1eda449b83ca199b49dece8b661973fab800dcef30ff079f3a58b818a10c" Sep 15 20:41:05.107201 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.107166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" event={"ID":"057261c3-367a-40b0-ba97-328f74076ee0","Type":"ContainerDied","Data":"343b1eda449b83ca199b49dece8b661973fab800dcef30ff079f3a58b818a10c"} Sep 15 20:41:05.109729 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.109383 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" event={"ID":"1e360961-0f5b-42b6-8a9b-576067c2e536","Type":"ContainerStarted","Data":"a2e78245ab7000e7b0de99070c9c2ca324d243c7fbdcc5c142484749afa3cda7"} Sep 15 20:41:05.136647 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.136590 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-27q29" podStartSLOduration=2.323419232 podStartE2EDuration="5.136573935s" totalImagesPullingTime="2.813154703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:05.133504512 +0000 UTC m=+254.263241099" watchObservedRunningTime="2026-09-15 20:41:05.136573935 +0000 UTC m=+254.266310517" Sep 15 20:41:05.185637 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.185559 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2c6tb" podStartSLOduration=2.42637517 podStartE2EDuration="5.185538659s" totalImagesPullingTime="2.759163489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:05.185346695 +0000 UTC m=+254.315083275" watchObservedRunningTime="2026-09-15 20:41:05.185538659 +0000 UTC m=+254.315275243" Sep 15 20:41:05.203338 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.203309 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9b9vt"] Sep 15 20:41:05.207344 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.206804 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.210962 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.210937 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d898r\"" Sep 15 20:41:05.211075 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.211039 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 20:41:05.211619 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.211599 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 20:41:05.216928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.216909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9b9vt"] Sep 15 20:41:05.302219 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.302189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a87410fa-9f76-4091-bbad-2a10be1cc046-crio-socket\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.302325 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.302222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.302325 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.302254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.302394 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.302339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgsb\" (UniqueName: \"kubernetes.io/projected/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-api-access-ptgsb\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.302394 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.302370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a87410fa-9f76-4091-bbad-2a10be1cc046-data-volume\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.323451 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.323377 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:05.402738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.402665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a87410fa-9f76-4091-bbad-2a10be1cc046-crio-socket\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.402738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.402713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.402792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a87410fa-9f76-4091-bbad-2a10be1cc046-crio-socket\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.402833 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.402868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.402893 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:05.902874849 +0000 UTC m=+255.032611421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.402926 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.402968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgsb\" (UniqueName: \"kubernetes.io/projected/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-api-access-ptgsb\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.402984 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:05.902967536 +0000 UTC m=+255.032704097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : configmap "kube-rbac-proxy" not found Sep 15 20:41:05.403097 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.403016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a87410fa-9f76-4091-bbad-2a10be1cc046-data-volume\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.403421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.403382 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a87410fa-9f76-4091-bbad-2a10be1cc046-data-volume\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.410389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.410365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgsb\" (UniqueName: \"kubernetes.io/projected/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-api-access-ptgsb\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.423325 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.423303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:05.610614 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.610583 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:05.669681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.669619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:05.854284 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.854255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:05.906387 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.906363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.906560 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.906426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:05.906560 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.906495 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:05.906560 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.906523 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.906560 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.906552 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:06.906536773 +0000 UTC m=+256.036273339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : configmap "kube-rbac-proxy" not found Sep 15 20:41:05.906724 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:05.906578 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:06.906561866 +0000 UTC m=+256.036298428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:05.915329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:05.915304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:06.016381 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.016350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:06.114779 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.114753 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/1.log" Sep 15 20:41:06.115195 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.115179 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/0.log" Sep 15 20:41:06.115243 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.115214 2580 generic.go:356] "Generic (PLEG): container finished" podID="057261c3-367a-40b0-ba97-328f74076ee0" containerID="d0174607821d818b68acf04cd48e96a388b5f78f75b150c1799c1c61da17a0e5" exitCode=255 Sep 15 20:41:06.115305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.115285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" event={"ID":"057261c3-367a-40b0-ba97-328f74076ee0","Type":"ContainerDied","Data":"d0174607821d818b68acf04cd48e96a388b5f78f75b150c1799c1c61da17a0e5"} Sep 15 20:41:06.115354 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.115342 2580 scope.go:117] "RemoveContainer" containerID="343b1eda449b83ca199b49dece8b661973fab800dcef30ff079f3a58b818a10c" Sep 15 20:41:06.115540 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.115525 2580 scope.go:117] "RemoveContainer" containerID="d0174607821d818b68acf04cd48e96a388b5f78f75b150c1799c1c61da17a0e5" Sep 15 20:41:06.115716 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:06.115689 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8zzs5_openshift-console-operator(057261c3-367a-40b0-ba97-328f74076ee0)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" podUID="057261c3-367a-40b0-ba97-328f74076ee0" Sep 15 20:41:06.116861 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.116712 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:06.317838 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.317738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:06.419558 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.419525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:06.914855 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.914824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:06.915017 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.914871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:06.915017 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:06.914982 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:06.915085 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:06.915012 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:06.915085 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:06.915048 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.915032351 +0000 UTC m=+258.044768915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:06.915085 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:06.915062 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.915056642 +0000 UTC m=+258.044793202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : configmap "kube-rbac-proxy" not found Sep 15 20:41:06.930762 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:06.930737 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ssxn_0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809/dns-node-resolver/0.log" Sep 15 20:41:07.120110 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:07.120080 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/1.log" Sep 15 20:41:07.120443 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:07.120429 2580 scope.go:117] "RemoveContainer" containerID="d0174607821d818b68acf04cd48e96a388b5f78f75b150c1799c1c61da17a0e5" Sep 15 20:41:07.120595 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:07.120577 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8zzs5_openshift-console-operator(057261c3-367a-40b0-ba97-328f74076ee0)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" podUID="057261c3-367a-40b0-ba97-328f74076ee0" Sep 15 20:41:08.083571 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.083533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:08.183507 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.183474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:08.243618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.243582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:08.330647 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.330618 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vjcfq_a11c1990-58c3-4921-84c7-f7bf86deaddf/node-ca/0.log" Sep 15 20:41:08.426953 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.426883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:08.426953 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.426950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:08.427102 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.426972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:08.427102 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.427038 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 20:41:08.427102 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.427046 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:16.42702471 +0000 UTC m=+265.556761277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : configmap references non-existent config key: service-ca.crt Sep 15 20:41:08.427102 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.427077 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 20:41:08.427102 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.427094 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:16.427082355 +0000 UTC m=+265.556818915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : secret "router-metrics-certs-default" not found Sep 15 20:41:08.427270 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.427108 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls podName:b4a56ae9-d999-4b36-9067-3e93bd4d4958 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:16.427099067 +0000 UTC m=+265.556835628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dgq29" (UID: "b4a56ae9-d999-4b36-9067-3e93bd4d4958") : secret "samples-operator-tls" not found Sep 15 20:41:08.528122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.528093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:08.528245 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.528224 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 20:41:08.528245 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.528241 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846b776d8-7vlwp: secret "image-registry-tls" not found Sep 15 20:41:08.528326 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.528281 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls podName:9ffc2beb-5319-45f1-851b-eafab7fd67a5 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:16.528269469 +0000 UTC m=+265.658006030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls") pod "image-registry-5846b776d8-7vlwp" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5") : secret "image-registry-tls" not found Sep 15 20:41:08.852735 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.852701 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:08.930604 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.930570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:08.930772 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.930692 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:08.930772 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.930692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:08.930877 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.930772 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:12.93075527 +0000 UTC m=+262.060491848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : configmap "kube-rbac-proxy" not found Sep 15 20:41:08.930877 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.930780 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:08.930877 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:08.930838 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:12.930821895 +0000 UTC m=+262.060558468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:08.931090 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:08.931071 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gnbcs_bd61f39f-8693-4da4-b81a-b84b3ade845f/serve-healthcheck-canary/0.log" Sep 15 20:41:11.018016 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:11.017984 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:11.018372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:11.018044 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:11.018372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:11.018335 2580 scope.go:117] "RemoveContainer" containerID="d0174607821d818b68acf04cd48e96a388b5f78f75b150c1799c1c61da17a0e5" Sep 15 20:41:11.018515 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:11.018495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8zzs5_openshift-console-operator(057261c3-367a-40b0-ba97-328f74076ee0)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" podUID="057261c3-367a-40b0-ba97-328f74076ee0" Sep 15 20:41:12.103032 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:12.103001 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:12.556645 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:12.556611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:12.733858 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:12.733825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:12.791731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:12.791704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:12.889742 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:12.889673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:12.963138 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:12.963111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:12.963257 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:12.963153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:12.963257 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:12.963226 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 20:41:12.963257 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:12.963250 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:12.963367 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:12.963277 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:20.96326395 +0000 UTC m=+270.093000511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : configmap "kube-rbac-proxy" not found Sep 15 20:41:12.963367 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:12.963297 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:20.963282959 +0000 UTC m=+270.093019519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:13.004863 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:13.004837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:13.093919 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:13.093894 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:13.252724 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:13.252693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:14.251039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:14.250965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:14.651460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:14.651357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:14.847380 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:14.847347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:15.048800 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:15.048767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:15.247590 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:15.247548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:15.465513 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:15.465430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:16.235604 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.235572 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:16.288578 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.288545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:16.445182 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.445150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:16.492354 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.492292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:16.492354 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.492348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:16.492527 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.492376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:16.492527 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:16.492475 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:32.492457918 +0000 UTC m=+281.622194486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : configmap references non-existent config key: service-ca.crt Sep 15 20:41:16.492527 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:16.492489 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 20:41:16.492629 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:16.492548 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs podName:4bbf573e-e3c2-4d9d-8711-5a6cf974c302 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:32.492533609 +0000 UTC m=+281.622270179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs") pod "router-default-59f448b5b8-4bm9g" (UID: "4bbf573e-e3c2-4d9d-8711-5a6cf974c302") : secret "router-metrics-certs-default" not found Sep 15 20:41:16.494660 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.494640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4a56ae9-d999-4b36-9067-3e93bd4d4958-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dgq29\" (UID: \"b4a56ae9-d999-4b36-9067-3e93bd4d4958\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:16.551248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.551224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:16.593527 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.593504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:16.595841 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.595821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"image-registry-5846b776d8-7vlwp\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:16.630982 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.630954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" Sep 15 20:41:16.710775 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.710748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:16.852276 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.852247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:16.895184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.895157 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29"] Sep 15 20:41:16.977152 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:16.977117 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5846b776d8-7vlwp"] Sep 15 20:41:16.980311 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:16.980276 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffc2beb_5319_45f1_851b_eafab7fd67a5.slice/crio-debd2f599852bb9d23f4e9490ff65db2bfb3d2a3a53fd3c327fe33e35f00f8e4 WatchSource:0}: Error finding container debd2f599852bb9d23f4e9490ff65db2bfb3d2a3a53fd3c327fe33e35f00f8e4: Status 404 returned error can't find the container with id debd2f599852bb9d23f4e9490ff65db2bfb3d2a3a53fd3c327fe33e35f00f8e4 Sep 15 20:41:17.050386 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:17.050363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:17.152741 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:17.152703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" event={"ID":"b4a56ae9-d999-4b36-9067-3e93bd4d4958","Type":"ContainerStarted","Data":"9fff305691a163b3f7c091df6bd94e60704dfb7b6fa32842a7050bc4c73cd741"} Sep 15 20:41:17.154110 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:17.154087 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" event={"ID":"9ffc2beb-5319-45f1-851b-eafab7fd67a5","Type":"ContainerStarted","Data":"d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc"} Sep 15 20:41:17.154239 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:17.154123 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" event={"ID":"9ffc2beb-5319-45f1-851b-eafab7fd67a5","Type":"ContainerStarted","Data":"debd2f599852bb9d23f4e9490ff65db2bfb3d2a3a53fd3c327fe33e35f00f8e4"} Sep 15 20:41:17.154239 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:17.154214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:17.172121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:17.171711 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" podStartSLOduration=17.171692596 podStartE2EDuration="17.171692596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:17.171127623 +0000 UTC m=+266.300864216" watchObservedRunningTime="2026-09-15 20:41:17.171692596 +0000 UTC m=+266.301429179" Sep 15 20:41:17.650050 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:17.650015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:18.245394 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:18.245365 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:18.388703 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:18.388670 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:18.488996 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:18.488955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:18.647645 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:18.647607 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:18.850332 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:18.850301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:19.120645 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:19.120614 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:19.163373 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:19.163287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" event={"ID":"b4a56ae9-d999-4b36-9067-3e93bd4d4958","Type":"ContainerStarted","Data":"41203c5d0abc373837270eda38a56b6cbf4abfa2609e8dbbb8d002c6cf5df086"} Sep 15 20:41:19.163373 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:19.163340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" event={"ID":"b4a56ae9-d999-4b36-9067-3e93bd4d4958","Type":"ContainerStarted","Data":"ac918566677f57174137b46028a30f5f591e8bf975f458c90adbb640fe08bd36"} Sep 15 20:41:19.180710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:19.180657 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dgq29" podStartSLOduration=17.256196518 podStartE2EDuration="19.180641279s" totalImagesPullingTime="1.924444761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:19.180366728 +0000 UTC m=+268.310103312" watchObservedRunningTime="2026-09-15 20:41:19.180641279 +0000 UTC m=+268.310377861" Sep 15 20:41:19.710305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:19.710270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:19.761064 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:19.761036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:19.845366 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:19.845334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:20.044521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:20.044492 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:20.450231 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:20.450157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:20.646443 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:20.646385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:21.031660 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:21.031626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:21.031809 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:21.031674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:21.031809 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:21.031770 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 20:41:21.031895 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:21.031834 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls podName:a87410fa-9f76-4091-bbad-2a10be1cc046 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:37.031819942 +0000 UTC m=+286.161556503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9b9vt" (UID: "a87410fa-9f76-4091-bbad-2a10be1cc046") : secret "insights-runtime-extractor-tls" not found Sep 15 20:41:21.032123 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:21.032107 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87410fa-9f76-4091-bbad-2a10be1cc046-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:21.050584 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:21.050550 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:21.653397 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:21.653363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:22.248643 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:22.248610 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:22.445852 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:22.445817 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:22.647610 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:22.647576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:23.048723 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:23.048690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:23.248791 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:23.248753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:24.307544 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:24.307516 2580 scope.go:117] "RemoveContainer" containerID="d0174607821d818b68acf04cd48e96a388b5f78f75b150c1799c1c61da17a0e5" Sep 15 20:41:25.185227 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:25.185195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:41:25.185711 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:25.185691 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/1.log" Sep 15 20:41:25.185835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:25.185730 2580 generic.go:356] "Generic (PLEG): container finished" podID="057261c3-367a-40b0-ba97-328f74076ee0" containerID="6328c5d13e26d7d4423987fca6ae7114b9f2018b767117bed0574d7da5b1fbf8" exitCode=255 Sep 15 20:41:25.185835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:25.185793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" event={"ID":"057261c3-367a-40b0-ba97-328f74076ee0","Type":"ContainerDied","Data":"6328c5d13e26d7d4423987fca6ae7114b9f2018b767117bed0574d7da5b1fbf8"} Sep 15 20:41:25.185941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:25.185852 2580 scope.go:117] "RemoveContainer" containerID="d0174607821d818b68acf04cd48e96a388b5f78f75b150c1799c1c61da17a0e5" Sep 15 20:41:25.186207 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:25.186180 2580 scope.go:117] "RemoveContainer" containerID="6328c5d13e26d7d4423987fca6ae7114b9f2018b767117bed0574d7da5b1fbf8" Sep 15 20:41:25.186441 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:25.186353 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-8zzs5_openshift-console-operator(057261c3-367a-40b0-ba97-328f74076ee0)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" podUID="057261c3-367a-40b0-ba97-328f74076ee0" Sep 15 20:41:26.190829 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:26.190801 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:41:28.848917 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:28.848880 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5846b776d8-7vlwp"] Sep 15 20:41:28.966573 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:28.966540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:29.080053 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:29.080017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:30.274247 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:30.274213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:30.332274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:30.332246 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:31.018104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:31.018062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:31.018298 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:31.018132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:31.018448 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:31.018435 2580 scope.go:117] "RemoveContainer" containerID="6328c5d13e26d7d4423987fca6ae7114b9f2018b767117bed0574d7da5b1fbf8" Sep 15 20:41:31.018593 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:31.018575 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-8zzs5_openshift-console-operator(057261c3-367a-40b0-ba97-328f74076ee0)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" podUID="057261c3-367a-40b0-ba97-328f74076ee0" Sep 15 20:41:31.853018 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:31.852977 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-25vtb" podUID="8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b" Sep 15 20:41:32.209829 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:32.209743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25vtb" Sep 15 20:41:32.518845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:32.518821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:32.518986 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:32.518882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:32.519386 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:32.519363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-service-ca-bundle\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:32.521116 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:32.521088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbf573e-e3c2-4d9d-8711-5a6cf974c302-metrics-certs\") pod \"router-default-59f448b5b8-4bm9g\" (UID: \"4bbf573e-e3c2-4d9d-8711-5a6cf974c302\") " pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:32.523981 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:32.523963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:32.791206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:32.791096 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59f448b5b8-4bm9g"] Sep 15 20:41:32.796010 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:32.795984 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbf573e_e3c2_4d9d_8711_5a6cf974c302.slice/crio-b970376a9248cd5927928e7d012ca9b8cbdc43f0d5a937276215972bb948e488 WatchSource:0}: Error finding container b970376a9248cd5927928e7d012ca9b8cbdc43f0d5a937276215972bb948e488: Status 404 returned error can't find the container with id b970376a9248cd5927928e7d012ca9b8cbdc43f0d5a937276215972bb948e488 Sep 15 20:41:33.214713 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:33.214613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" event={"ID":"4bbf573e-e3c2-4d9d-8711-5a6cf974c302","Type":"ContainerStarted","Data":"87a8df631faad82746bf43e3e07d807e28f368db1490e487e9fc2c91d61b5f3a"} Sep 15 20:41:33.214713 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:33.214682 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" event={"ID":"4bbf573e-e3c2-4d9d-8711-5a6cf974c302","Type":"ContainerStarted","Data":"b970376a9248cd5927928e7d012ca9b8cbdc43f0d5a937276215972bb948e488"} Sep 15 20:41:33.233022 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:33.232974 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" podStartSLOduration=33.232960491 podStartE2EDuration="33.232960491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:33.231744406 +0000 UTC m=+282.361481030" watchObservedRunningTime="2026-09-15 20:41:33.232960491 +0000 UTC m=+282.362697074" Sep 15 20:41:33.524439 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:33.524400 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:33.526701 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:33.526683 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:34.217690 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:34.217659 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:34.218865 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:34.218842 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59f448b5b8-4bm9g" Sep 15 20:41:34.794700 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:34.794640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:34.894618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:34.894540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:34.964010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:34.963977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:35.170524 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:35.170444 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:35.741071 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:35.741032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:41:35.743301 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:35.743281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b-metrics-tls\") pod \"dns-default-25vtb\" (UID: \"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b\") " pod="openshift-dns/dns-default-25vtb" Sep 15 20:41:35.813872 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:35.813842 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx25c\"" Sep 15 20:41:35.821766 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:35.821741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25vtb" Sep 15 20:41:36.089569 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:36.089535 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25vtb"] Sep 15 20:41:36.093066 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:36.093038 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff12cc4_90d7_46ce_83d4_6ceb6a9dc77b.slice/crio-c5f5ebfbccf70f70a48433378211b87e3c9235c646c1276f88bfeec7d0a173bc WatchSource:0}: Error finding container c5f5ebfbccf70f70a48433378211b87e3c9235c646c1276f88bfeec7d0a173bc: Status 404 returned error can't find the container with id c5f5ebfbccf70f70a48433378211b87e3c9235c646c1276f88bfeec7d0a173bc Sep 15 20:41:36.224239 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:36.224206 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25vtb" event={"ID":"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b","Type":"ContainerStarted","Data":"c5f5ebfbccf70f70a48433378211b87e3c9235c646c1276f88bfeec7d0a173bc"} Sep 15 20:41:37.052321 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:37.052289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:37.054718 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:37.054690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87410fa-9f76-4091-bbad-2a10be1cc046-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9b9vt\" (UID: \"a87410fa-9f76-4091-bbad-2a10be1cc046\") " pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:37.317564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:37.317483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9b9vt" Sep 15 20:41:37.808318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:37.808291 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9b9vt"] Sep 15 20:41:37.867186 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:37.867114 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87410fa_9f76_4091_bbad_2a10be1cc046.slice/crio-060c599779364ae7dcd6af51772d998b488bc17448163526051d2f4f08bbbfd2 WatchSource:0}: Error finding container 060c599779364ae7dcd6af51772d998b488bc17448163526051d2f4f08bbbfd2: Status 404 returned error can't find the container with id 060c599779364ae7dcd6af51772d998b488bc17448163526051d2f4f08bbbfd2 Sep 15 20:41:37.965424 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:37.965386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:38.066561 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.066538 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:38.237708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.237669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25vtb" event={"ID":"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b","Type":"ContainerStarted","Data":"13549ddaf446f3edba836b7062ebfe952cb8e3753c5432f77051fdcbd4be32a0"} Sep 15 20:41:38.237867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.237760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25vtb" event={"ID":"8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b","Type":"ContainerStarted","Data":"44846a34d46329d51a9cd5e7af7a9fc9e8ab53755f5356368ff86ac861a5590f"} Sep 15 20:41:38.237867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.237790 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-25vtb" Sep 15 20:41:38.239335 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.239314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9b9vt" event={"ID":"a87410fa-9f76-4091-bbad-2a10be1cc046","Type":"ContainerStarted","Data":"16337ec4a2273b1e5dc7769233477521a3c5149948911d78b0b4daf819b9ef66"} Sep 15 20:41:38.239450 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.239354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9b9vt" event={"ID":"a87410fa-9f76-4091-bbad-2a10be1cc046","Type":"ContainerStarted","Data":"060c599779364ae7dcd6af51772d998b488bc17448163526051d2f4f08bbbfd2"} Sep 15 20:41:38.255676 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.255635 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-25vtb" podStartSLOduration=251.448286837 podStartE2EDuration="4m13.255623064s" totalImagesPullingTime="1.807336227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:38.25398659 +0000 UTC m=+287.383723173" watchObservedRunningTime="2026-09-15 20:41:38.255623064 +0000 UTC m=+287.385359689" Sep 15 20:41:38.808293 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.808259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:38.854822 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.854793 2580 patch_prober.go:28] interesting pod/image-registry-5846b776d8-7vlwp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 15 20:41:38.854964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:38.854842 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" podUID="9ffc2beb-5319-45f1-851b-eafab7fd67a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 20:41:39.244279 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:39.244191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9b9vt" event={"ID":"a87410fa-9f76-4091-bbad-2a10be1cc046","Type":"ContainerStarted","Data":"307e7008ed0002c4503766908c279dc58d74cfb97301a58cb91182b22873daea"} Sep 15 20:41:41.253756 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:41.253721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9b9vt" event={"ID":"a87410fa-9f76-4091-bbad-2a10be1cc046","Type":"ContainerStarted","Data":"04655989efa889522f9e02b996ebb89e3750e9788346e73a5e6617f6a68202e2"} Sep 15 20:41:41.270038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:41.269989 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9b9vt" podStartSLOduration=33.841862581 podStartE2EDuration="36.269972382s" totalImagesPullingTime="2.428109801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:41:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:41.268248018 +0000 UTC m=+290.397984601" watchObservedRunningTime="2026-09-15 20:41:41.269972382 +0000 UTC m=+290.399708965" Sep 15 20:41:41.889238 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:41.889195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:42.286179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:42.286135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:42.307905 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:42.307882 2580 scope.go:117] "RemoveContainer" containerID="6328c5d13e26d7d4423987fca6ae7114b9f2018b767117bed0574d7da5b1fbf8" Sep 15 20:41:42.308027 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:42.308011 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-8zzs5_openshift-console-operator(057261c3-367a-40b0-ba97-328f74076ee0)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" podUID="057261c3-367a-40b0-ba97-328f74076ee0" Sep 15 20:41:42.387375 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:42.387344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:42.438337 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:42.438314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.082230 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.082194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.195160 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.195129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.297659 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.297632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.323362 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.323328 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j"] Sep 15 20:41:44.326907 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.326885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.329418 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.329384 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 15 20:41:44.329663 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.329646 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zhs6s\"" Sep 15 20:41:44.330249 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.330234 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 20:41:44.330549 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.330536 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 15 20:41:44.330787 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.330755 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 20:41:44.330954 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.330938 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 20:41:44.338736 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.338717 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j"] Sep 15 20:41:44.340059 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.340037 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn"] Sep 15 20:41:44.342206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.342190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.344077 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.344025 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 15 20:41:44.344077 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.344056 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vjtwj\"" Sep 15 20:41:44.344263 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.344246 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 15 20:41:44.344321 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.344259 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 15 20:41:44.349222 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.349202 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fq67k"] Sep 15 20:41:44.351057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.351041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.352940 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.352920 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 20:41:44.353042 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.353027 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 20:41:44.353121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.353107 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 20:41:44.353252 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.353239 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qq8fj\"" Sep 15 20:41:44.355895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.355877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn"] Sep 15 20:41:44.377316 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.377263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.408876 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.408856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvmj\" (UniqueName: \"kubernetes.io/projected/0b05402b-6f75-451a-8bfe-296666a116c9-kube-api-access-7dvmj\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.408964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.408888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.408964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.408905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682mb\" (UniqueName: \"kubernetes.io/projected/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-api-access-682mb\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.408964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.408922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-root\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.408961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b05402b-6f75-451a-8bfe-296666a116c9-metrics-client-ca\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.408988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-wtmp\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-sys\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.409074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jct8m\" (UniqueName: \"kubernetes.io/projected/5453c7e3-cd47-426b-8f3c-b8b88b555b79-kube-api-access-jct8m\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.409245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5453c7e3-cd47-426b-8f3c-b8b88b555b79-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.409245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-textfile\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.409245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-tls\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.409245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.409245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.409472 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.409472 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.409330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.440723 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.440700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.510229 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.510346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvmj\" (UniqueName: \"kubernetes.io/projected/0b05402b-6f75-451a-8bfe-296666a116c9-kube-api-access-7dvmj\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.510346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682mb\" (UniqueName: \"kubernetes.io/projected/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-api-access-682mb\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.510346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-root\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b05402b-6f75-451a-8bfe-296666a116c9-metrics-client-ca\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-wtmp\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-sys\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-root\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jct8m\" (UniqueName: \"kubernetes.io/projected/5453c7e3-cd47-426b-8f3c-b8b88b555b79-kube-api-access-jct8m\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5453c7e3-cd47-426b-8f3c-b8b88b555b79-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-sys\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-textfile\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-tls\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.510641 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.511248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-wtmp\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.511248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.510669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.511248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511097 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.511248 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:44.511190 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 15 20:41:44.511248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.511507 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:44.511257 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-tls podName:5453c7e3-cd47-426b-8f3c-b8b88b555b79 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:45.011237158 +0000 UTC m=+294.140973750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-swb7j" (UID: "5453c7e3-cd47-426b-8f3c-b8b88b555b79") : secret "openshift-state-metrics-tls" not found Sep 15 20:41:44.511507 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b05402b-6f75-451a-8bfe-296666a116c9-metrics-client-ca\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.511507 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5453c7e3-cd47-426b-8f3c-b8b88b555b79-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.511507 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.511678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.511678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.511843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-textfile\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.511924 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.511889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.513130 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.513094 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.513275 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.513258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.513741 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.513721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.513849 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.513833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-tls\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.513975 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.513961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b05402b-6f75-451a-8bfe-296666a116c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.517591 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.517565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvmj\" (UniqueName: \"kubernetes.io/projected/0b05402b-6f75-451a-8bfe-296666a116c9-kube-api-access-7dvmj\") pod \"node-exporter-fq67k\" (UID: \"0b05402b-6f75-451a-8bfe-296666a116c9\") " pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.519537 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.519515 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jct8m\" (UniqueName: \"kubernetes.io/projected/5453c7e3-cd47-426b-8f3c-b8b88b555b79-kube-api-access-jct8m\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:44.519948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.519932 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-682mb\" (UniqueName: \"kubernetes.io/projected/3b963fbb-4fc0-4324-afeb-a43b23a5fb06-kube-api-access-682mb\") pod \"kube-state-metrics-59fbdd77d-rzhwn\" (UID: \"3b963fbb-4fc0-4324-afeb-a43b23a5fb06\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.544352 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.544325 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.651393 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.651342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" Sep 15 20:41:44.659681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.659661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fq67k" Sep 15 20:41:44.669306 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:44.669279 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b05402b_6f75_451a_8bfe_296666a116c9.slice/crio-41a5c09b837881b38c5c46c14d60523680af3d1b9f4973a2d9a4c68aeded7806 WatchSource:0}: Error finding container 41a5c09b837881b38c5c46c14d60523680af3d1b9f4973a2d9a4c68aeded7806: Status 404 returned error can't find the container with id 41a5c09b837881b38c5c46c14d60523680af3d1b9f4973a2d9a4c68aeded7806 Sep 15 20:41:44.728445 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.728379 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.891626 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.891597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:44.921615 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:44.921547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn"] Sep 15 20:41:44.924705 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:44.924677 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b963fbb_4fc0_4324_afeb_a43b23a5fb06.slice/crio-b7a8868ece34ec61e49956fe7f16da4c2e0a94b102c74735280f0b8228b17a34 WatchSource:0}: Error finding container b7a8868ece34ec61e49956fe7f16da4c2e0a94b102c74735280f0b8228b17a34: Status 404 returned error can't find the container with id b7a8868ece34ec61e49956fe7f16da4c2e0a94b102c74735280f0b8228b17a34 Sep 15 20:41:45.015079 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.015052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:45.017341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.017300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5453c7e3-cd47-426b-8f3c-b8b88b555b79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-swb7j\" (UID: \"5453c7e3-cd47-426b-8f3c-b8b88b555b79\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:45.236791 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.236718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" Sep 15 20:41:45.269421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.269373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq67k" event={"ID":"0b05402b-6f75-451a-8bfe-296666a116c9","Type":"ContainerStarted","Data":"41a5c09b837881b38c5c46c14d60523680af3d1b9f4973a2d9a4c68aeded7806"} Sep 15 20:41:45.270915 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.270885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" event={"ID":"3b963fbb-4fc0-4324-afeb-a43b23a5fb06","Type":"ContainerStarted","Data":"b7a8868ece34ec61e49956fe7f16da4c2e0a94b102c74735280f0b8228b17a34"} Sep 15 20:41:45.303747 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.303720 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:45.385732 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.385694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:45.390885 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.390863 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:41:45.393381 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.393364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.395343 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.395326 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nbc8f\"" Sep 15 20:41:45.395918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.395902 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 15 20:41:45.395982 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.395958 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 15 20:41:45.395982 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.395968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 15 20:41:45.396118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.396102 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 15 20:41:45.396184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.396158 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 15 20:41:45.396184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.396104 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 15 20:41:45.396300 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.396198 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 15 20:41:45.396300 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.396205 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 15 20:41:45.396300 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.396207 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 15 20:41:45.413821 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.413798 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:41:45.417740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.417828 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.417828 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.417828 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.417918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.417918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.417982 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-config-out\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.417982 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.418040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.417988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.418040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.418003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-web-config\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.418040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.418024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbv96\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-kube-api-access-pbv96\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.418132 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.418101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.418184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.418170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.505577 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.505541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:45.512912 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.512889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j"] Sep 15 20:41:45.515809 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:45.515771 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5453c7e3_cd47_426b_8f3c_b8b88b555b79.slice/crio-806677c70fd818c716019dbc5b7502d501b5c4a145cf4c6108f2e5a713ef2888 WatchSource:0}: Error finding container 806677c70fd818c716019dbc5b7502d501b5c4a145cf4c6108f2e5a713ef2888: Status 404 returned error can't find the container with id 806677c70fd818c716019dbc5b7502d501b5c4a145cf4c6108f2e5a713ef2888 Sep 15 20:41:45.518698 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518698 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518817 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518817 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518817 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518961 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518961 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518961 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.518961 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-config-out\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.519161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.519161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.518994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.519161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.519023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-web-config\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.519161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.519061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbv96\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-kube-api-access-pbv96\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.520792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.519540 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.520792 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:45.519609 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle podName:07f7977a-73ac-4881-b12f-05646ea233f7 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:46.019589311 +0000 UTC m=+295.149325876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7") : configmap references non-existent config key: ca-bundle.crt Sep 15 20:41:45.520792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.519907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.522247 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.522206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.522350 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.522304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.524560 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.524535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.524658 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.524599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.525044 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.525022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.525323 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.525285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.525812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.525779 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.526045 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.526008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-config-out\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.526146 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.526049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-web-config\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.530458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.530365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbv96\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-kube-api-access-pbv96\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:45.605274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:45.605242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:46.024577 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.024494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:46.025499 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.025470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:46.277259 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.277172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" event={"ID":"5453c7e3-cd47-426b-8f3c-b8b88b555b79","Type":"ContainerStarted","Data":"42877a26904e29fc7243dae0572ae79b56ae9dc6d739e92ac6118e345711d8dc"} Sep 15 20:41:46.277715 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.277285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" event={"ID":"5453c7e3-cd47-426b-8f3c-b8b88b555b79","Type":"ContainerStarted","Data":"8e33b8a4abf3f48d0e9cfc7ced6dc1715c0fbd6a8f44806e387f90d92f02de56"} Sep 15 20:41:46.277715 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.277321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" event={"ID":"5453c7e3-cd47-426b-8f3c-b8b88b555b79","Type":"ContainerStarted","Data":"806677c70fd818c716019dbc5b7502d501b5c4a145cf4c6108f2e5a713ef2888"} Sep 15 20:41:46.279064 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.279035 2580 generic.go:356] "Generic (PLEG): container finished" podID="0b05402b-6f75-451a-8bfe-296666a116c9" containerID="ca944cac7edad46d3c20e81e1689041c4c8ac3c6284ae44de39b458c2556deb6" exitCode=0 Sep 15 20:41:46.279176 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.279097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq67k" event={"ID":"0b05402b-6f75-451a-8bfe-296666a116c9","Type":"ContainerDied","Data":"ca944cac7edad46d3c20e81e1689041c4c8ac3c6284ae44de39b458c2556deb6"} Sep 15 20:41:46.303136 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.303112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:41:46.593314 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:46.593208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:41:46.790334 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:46.790298 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f7977a_73ac_4881_b12f_05646ea233f7.slice/crio-53f89a783cc82a58bf6a69091badcc5c3c0da1aea46766b0480e0e38a15f95ee WatchSource:0}: Error finding container 53f89a783cc82a58bf6a69091badcc5c3c0da1aea46766b0480e0e38a15f95ee: Status 404 returned error can't find the container with id 53f89a783cc82a58bf6a69091badcc5c3c0da1aea46766b0480e0e38a15f95ee Sep 15 20:41:47.239899 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.239850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:47.285496 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.285464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq67k" event={"ID":"0b05402b-6f75-451a-8bfe-296666a116c9","Type":"ContainerStarted","Data":"8ff9cb383ea48b834b5c8319d56b61600c45152125328c3795e47db2af766463"} Sep 15 20:41:47.285848 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.285512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq67k" event={"ID":"0b05402b-6f75-451a-8bfe-296666a116c9","Type":"ContainerStarted","Data":"be3f506eeafcdb66707128f2ce35dbd5f99d777fb6e2b00fe047ab85ebabbed4"} Sep 15 20:41:47.288121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.288096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" event={"ID":"3b963fbb-4fc0-4324-afeb-a43b23a5fb06","Type":"ContainerStarted","Data":"de2d95a6716413596518e13405df3656a05614dfaecad705e3d40a4ca76c94e8"} Sep 15 20:41:47.288274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.288139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" event={"ID":"3b963fbb-4fc0-4324-afeb-a43b23a5fb06","Type":"ContainerStarted","Data":"61ca03e8fa1a36646daaf650244f49cbc40174cbb9761ab7ff9dd348dfd7246c"} Sep 15 20:41:47.288274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.288161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" event={"ID":"3b963fbb-4fc0-4324-afeb-a43b23a5fb06","Type":"ContainerStarted","Data":"4f1427ca1884f1bcbc51f10a3d2a15dae68b6bc83850ad6a9355a2fea7442961"} Sep 15 20:41:47.290534 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.290512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" event={"ID":"5453c7e3-cd47-426b-8f3c-b8b88b555b79","Type":"ContainerStarted","Data":"9e34910c6b663a02ba393c52f40fc56a050862a64e2db53743025530c8d0837f"} Sep 15 20:41:47.291777 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.291759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerStarted","Data":"53f89a783cc82a58bf6a69091badcc5c3c0da1aea46766b0480e0e38a15f95ee"} Sep 15 20:41:47.308426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.308358 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fq67k" podStartSLOduration=2.115207265 podStartE2EDuration="3.308341045s" totalImagesPullingTime="1.133013289s" totalInitContainerRuntime="60.120491ms" isStatefulPod=true podCreationTimestamp="2026-09-15 20:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:47.306773686 +0000 UTC m=+296.436510307" watchObservedRunningTime="2026-09-15 20:41:47.308341045 +0000 UTC m=+296.438077610" Sep 15 20:41:47.313088 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.313067 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-898449498-9xrh5"] Sep 15 20:41:47.318193 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.316830 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.319213 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.318987 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 15 20:41:47.319297 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.319266 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 15 20:41:47.319549 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.319533 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 15 20:41:47.319739 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.319722 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 15 20:41:47.319859 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.319840 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 15 20:41:47.320011 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.319768 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2bcjqmtvr7or2\"" Sep 15 20:41:47.320259 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.320243 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8w8dm\"" Sep 15 20:41:47.323549 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.323519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:47.327813 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.327775 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-898449498-9xrh5"] Sep 15 20:41:47.331169 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.331123 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-swb7j" podStartSLOduration=2.005009537 podStartE2EDuration="3.331105635s" totalImagesPullingTime="1.326096098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:47.329667709 +0000 UTC m=+296.459404293" watchObservedRunningTime="2026-09-15 20:41:47.331105635 +0000 UTC m=+296.460842222" Sep 15 20:41:47.350258 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.348270 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rzhwn" podStartSLOduration=1.44063339 podStartE2EDuration="3.348252005s" totalImagesPullingTime="1.907618615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:47.345257349 +0000 UTC m=+296.474993989" watchObservedRunningTime="2026-09-15 20:41:47.348252005 +0000 UTC m=+296.477988590" Sep 15 20:41:47.412358 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.412279 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:47.438498 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdp8\" (UniqueName: \"kubernetes.io/projected/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-kube-api-access-ctdp8\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.438606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.438606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-grpc-tls\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.438606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.438786 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.438786 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-metrics-client-ca\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.438786 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-tls\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.438951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.438789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.539804 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.539775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.539937 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.539826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-metrics-client-ca\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.539937 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.539847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-tls\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.539937 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.539901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.540027 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.539947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdp8\" (UniqueName: \"kubernetes.io/projected/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-kube-api-access-ctdp8\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.540124 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.540095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.540187 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.540158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-grpc-tls\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.540234 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.540200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.540712 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.540647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-metrics-client-ca\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.542593 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.542562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.542721 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.542702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-grpc-tls\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.542785 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.542742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.542843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.542785 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.542941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.542921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.542992 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.542946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-secret-thanos-querier-tls\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.547446 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.547426 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdp8\" (UniqueName: \"kubernetes.io/projected/c1bf3427-37b9-4d9f-ad60-bbf6fbf21072-kube-api-access-ctdp8\") pod \"thanos-querier-898449498-9xrh5\" (UID: \"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072\") " pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.629333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.629298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:47.798390 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.798351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:47.907222 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:47.907188 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-898449498-9xrh5"] Sep 15 20:41:47.910862 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:41:47.910835 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bf3427_37b9_4d9f_ad60_bbf6fbf21072.slice/crio-675aa60f92878d8aab0d260870525b79a809c767d0d374ec88301a51dac941bf WatchSource:0}: Error finding container 675aa60f92878d8aab0d260870525b79a809c767d0d374ec88301a51dac941bf: Status 404 returned error can't find the container with id 675aa60f92878d8aab0d260870525b79a809c767d0d374ec88301a51dac941bf Sep 15 20:41:48.246602 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:48.246524 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-25vtb" Sep 15 20:41:48.298126 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:48.298091 2580 generic.go:356] "Generic (PLEG): container finished" podID="07f7977a-73ac-4881-b12f-05646ea233f7" containerID="2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19" exitCode=0 Sep 15 20:41:48.298604 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:48.298126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19"} Sep 15 20:41:48.299645 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:48.299617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" event={"ID":"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072","Type":"ContainerStarted","Data":"675aa60f92878d8aab0d260870525b79a809c767d0d374ec88301a51dac941bf"} Sep 15 20:41:48.309053 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:48.309021 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5vvbb" podUID="dad0237e-3438-4e30-a3f8-5f76d25d0763" Sep 15 20:41:48.853837 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:48.853811 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:49.106338 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:49.106245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:49.310337 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:49.310269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:50.376339 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:50.376305 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:50.471243 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:50.471217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:50.583702 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:50.583674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:51.204340 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.204306 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:41:51.204739 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.204717 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:41:51.223213 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.223188 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 20:41:51.317297 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.317266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerStarted","Data":"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f"} Sep 15 20:41:51.319431 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.317337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerStarted","Data":"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97"} Sep 15 20:41:51.319431 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.317379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerStarted","Data":"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0"} Sep 15 20:41:51.319431 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.317427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerStarted","Data":"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099"} Sep 15 20:41:51.319431 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.317450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerStarted","Data":"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f"} Sep 15 20:41:51.319618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.319436 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" event={"ID":"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072","Type":"ContainerStarted","Data":"73a86df4de1c5da285e4b20f396158d7b3aa1eeee61686669f1cb156ac20ec5d"} Sep 15 20:41:51.319618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.319497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" event={"ID":"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072","Type":"ContainerStarted","Data":"7d5dbe3bc376f032f177a50ec90638ab47c96c148b278282bbe96ddff75aaaff"} Sep 15 20:41:51.319618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.319534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" event={"ID":"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072","Type":"ContainerStarted","Data":"47f49cd8367cada6a428ac8840d90da9e10fc2ee2a58afcd06228240f84e464c"} Sep 15 20:41:51.663417 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:51.663374 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:52.327166 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.327128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" event={"ID":"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072","Type":"ContainerStarted","Data":"b2d01015f5d438fda3bcb798366f2921f7ac18c60f792ca433173abe5013ba19"} Sep 15 20:41:52.327365 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.327235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" event={"ID":"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072","Type":"ContainerStarted","Data":"296844b50a5712448cf991fdfcb4bace4092002463f5724cc5e89421fbcfa999"} Sep 15 20:41:52.327365 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.327266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" event={"ID":"c1bf3427-37b9-4d9f-ad60-bbf6fbf21072","Type":"ContainerStarted","Data":"2e86a6876a3ed85e79809a86acb3a3e37f5a9941470a199fc4222eca9c89c41a"} Sep 15 20:41:52.327365 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.327298 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:52.330460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.330435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerStarted","Data":"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f"} Sep 15 20:41:52.349334 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.349293 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" podStartSLOduration=2.051863824 podStartE2EDuration="5.349280168s" totalImagesPullingTime="3.297416344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:52.348068987 +0000 UTC m=+301.477805574" watchObservedRunningTime="2026-09-15 20:41:52.349280168 +0000 UTC m=+301.479016751" Sep 15 20:41:52.375726 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.375680 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.255374916 podStartE2EDuration="7.375666626s" totalImagesPullingTime="4.105176236s" totalInitContainerRuntime="15.115474ms" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:52.374019282 +0000 UTC m=+301.503755866" watchObservedRunningTime="2026-09-15 20:41:52.375666626 +0000 UTC m=+301.505403210" Sep 15 20:41:52.497349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.497316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:52.550720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:52.550692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:53.867180 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:53.867133 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" podUID="9ffc2beb-5319-45f1-851b-eafab7fd67a5" containerName="registry" containerID="cri-o://d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc" gracePeriod=30 Sep 15 20:41:54.105774 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.105751 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:54.198453 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198387 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-installation-pull-secrets\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198453 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198436 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198464 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzc69\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-kube-api-access-kzc69\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198483 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-certificates\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198500 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-bound-sa-token\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198547 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-image-registry-private-configuration\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198569 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ffc2beb-5319-45f1-851b-eafab7fd67a5-ca-trust-extracted\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198604 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-trusted-ca\") pod \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\" (UID: \"9ffc2beb-5319-45f1-851b-eafab7fd67a5\") " Sep 15 20:41:54.198965 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.198926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:41:54.199208 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.199182 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:41:54.201040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.200980 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:41:54.201040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.200998 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:41:54.201040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.200985 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:41:54.201201 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.201045 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-kube-api-access-kzc69" (OuterVolumeSpecName: "kube-api-access-kzc69") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "kube-api-access-kzc69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:41:54.201276 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.201260 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:41:54.206834 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.206809 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffc2beb-5319-45f1-851b-eafab7fd67a5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9ffc2beb-5319-45f1-851b-eafab7fd67a5" (UID: "9ffc2beb-5319-45f1-851b-eafab7fd67a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:41:54.299739 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299716 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-installation-pull-secrets\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.299739 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299736 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.299866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299745 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzc69\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-kube-api-access-kzc69\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.299866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299754 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-registry-certificates\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.299866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299762 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ffc2beb-5319-45f1-851b-eafab7fd67a5-bound-sa-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.299866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299770 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ffc2beb-5319-45f1-851b-eafab7fd67a5-image-registry-private-configuration\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.299866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299778 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ffc2beb-5319-45f1-851b-eafab7fd67a5-ca-trust-extracted\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.299866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.299786 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2beb-5319-45f1-851b-eafab7fd67a5-trusted-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:41:54.307942 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.307925 2580 scope.go:117] "RemoveContainer" containerID="6328c5d13e26d7d4423987fca6ae7114b9f2018b767117bed0574d7da5b1fbf8" Sep 15 20:41:54.338516 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.338494 2580 generic.go:356] "Generic (PLEG): container finished" podID="9ffc2beb-5319-45f1-851b-eafab7fd67a5" containerID="d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc" exitCode=0 Sep 15 20:41:54.338606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.338526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" event={"ID":"9ffc2beb-5319-45f1-851b-eafab7fd67a5","Type":"ContainerDied","Data":"d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc"} Sep 15 20:41:54.338606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.338545 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" Sep 15 20:41:54.338606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.338581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846b776d8-7vlwp" event={"ID":"9ffc2beb-5319-45f1-851b-eafab7fd67a5","Type":"ContainerDied","Data":"debd2f599852bb9d23f4e9490ff65db2bfb3d2a3a53fd3c327fe33e35f00f8e4"} Sep 15 20:41:54.338752 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.338609 2580 scope.go:117] "RemoveContainer" containerID="d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc" Sep 15 20:41:54.347944 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.347929 2580 scope.go:117] "RemoveContainer" containerID="d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc" Sep 15 20:41:54.348227 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:41:54.348204 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc\": container with ID starting with d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc not found: ID does not exist" containerID="d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc" Sep 15 20:41:54.348295 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.348237 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc"} err="failed to get container status \"d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc\": rpc error: code = NotFound desc = could not find container \"d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc\": container with ID starting with d925f52e974da99eb48311b41ba564c14e2fbeb769e24725b519932eb0a7e4bc not found: ID does not exist" Sep 15 20:41:54.359464 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.359443 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5846b776d8-7vlwp"] Sep 15 20:41:54.361622 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:54.361604 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5846b776d8-7vlwp"] Sep 15 20:41:55.221182 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.221146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:55.312932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.312898 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffc2beb-5319-45f1-851b-eafab7fd67a5" path="/var/lib/kubelet/pods/9ffc2beb-5319-45f1-851b-eafab7fd67a5/volumes" Sep 15 20:41:55.321555 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.321530 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:55.345253 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.345234 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:41:55.345334 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.345279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" event={"ID":"057261c3-367a-40b0-ba97-328f74076ee0","Type":"ContainerStarted","Data":"cd8c02630a09d640e634735170e54b9faff2a52cc805f1d23793963dfe5c22ce"} Sep 15 20:41:55.345502 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.345486 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:55.350055 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.350037 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" Sep 15 20:41:55.365237 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.365199 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-8zzs5" podStartSLOduration=52.53537363 podStartE2EDuration="55.365186454s" totalImagesPullingTime="2.829812824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:55.364212387 +0000 UTC m=+304.493948972" watchObservedRunningTime="2026-09-15 20:41:55.365186454 +0000 UTC m=+304.494923063" Sep 15 20:41:55.422172 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.422142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:55.521870 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.521847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:55.622338 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.622308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:55.723845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.723814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:55.892835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:55.892766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:56.297126 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:56.297094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:56.398490 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:56.398458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:56.687919 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:56.687842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:56.898100 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:56.898073 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:57.280332 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:57.280304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:57.505180 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:57.505151 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:58.340838 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:58.340812 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-898449498-9xrh5" Sep 15 20:41:59.303089 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:59.303056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:41:59.403615 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:41:59.403580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:02.318425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:02.318383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:02.385231 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:02.385202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:03.308090 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:03.308054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:42:04.110470 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:04.110387 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:04.162233 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:04.162200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:06.117329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:06.117294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:06.215209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:06.215174 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:10.508986 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.508952 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7599dbb675-654xh"] Sep 15 20:42:10.509349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.509270 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffc2beb-5319-45f1-851b-eafab7fd67a5" containerName="registry" Sep 15 20:42:10.509349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.509281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffc2beb-5319-45f1-851b-eafab7fd67a5" containerName="registry" Sep 15 20:42:10.509349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.509338 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffc2beb-5319-45f1-851b-eafab7fd67a5" containerName="registry" Sep 15 20:42:10.511941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.511920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.513967 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.513943 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 15 20:42:10.514107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.513945 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 15 20:42:10.514107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.514071 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 15 20:42:10.514740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.514713 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:10.514864 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.514835 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 15 20:42:10.514930 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.514861 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sjxrh\"" Sep 15 20:42:10.514930 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.514883 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 15 20:42:10.515031 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.514948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 15 20:42:10.515270 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.515251 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 15 20:42:10.520112 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.520090 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 15 20:42:10.521841 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.521802 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7599dbb675-654xh"] Sep 15 20:42:10.582780 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.582755 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:10.632618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.632596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-serving-cert\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.632704 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.632626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsl8m\" (UniqueName: \"kubernetes.io/projected/932a9a20-ca3f-4d8b-990c-f43a778f6b94-kube-api-access-bsl8m\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.632704 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.632648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-oauth-serving-cert\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.632775 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.632729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-trusted-ca-bundle\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.632775 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.632768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-service-ca\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.632840 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.632797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-oauth-config\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.632875 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.632844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-config\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.733869 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.733845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-oauth-config\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.733957 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.733892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-config\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.733957 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.733916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-serving-cert\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.734046 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.734026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsl8m\" (UniqueName: \"kubernetes.io/projected/932a9a20-ca3f-4d8b-990c-f43a778f6b94-kube-api-access-bsl8m\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.734081 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.734068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-oauth-serving-cert\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.734137 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.734123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-trusted-ca-bundle\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.734188 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.734167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-service-ca\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.734724 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.734698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-config\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.734809 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.734740 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-oauth-serving-cert\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.734850 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.734817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-service-ca\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.735139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.735117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-trusted-ca-bundle\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.736576 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.736549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-serving-cert\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.736676 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.736613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-oauth-config\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.743706 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.743686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsl8m\" (UniqueName: \"kubernetes.io/projected/932a9a20-ca3f-4d8b-990c-f43a778f6b94-kube-api-access-bsl8m\") pod \"console-7599dbb675-654xh\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:10.823005 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:10.822952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:11.089662 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:11.089593 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7599dbb675-654xh"] Sep 15 20:42:11.092499 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:42:11.092470 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932a9a20_ca3f_4d8b_990c_f43a778f6b94.slice/crio-d0858c3ce3e1b059cfb1c41d950f018d94d4dc868cc3227b7a32c573bdb01a06 WatchSource:0}: Error finding container d0858c3ce3e1b059cfb1c41d950f018d94d4dc868cc3227b7a32c573bdb01a06: Status 404 returned error can't find the container with id d0858c3ce3e1b059cfb1c41d950f018d94d4dc868cc3227b7a32c573bdb01a06 Sep 15 20:42:11.094726 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:11.094710 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:42:11.393792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:11.393700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:11.404122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:11.404094 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7599dbb675-654xh" event={"ID":"932a9a20-ca3f-4d8b-990c-f43a778f6b94","Type":"ContainerStarted","Data":"d0858c3ce3e1b059cfb1c41d950f018d94d4dc868cc3227b7a32c573bdb01a06"} Sep 15 20:42:11.452858 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:11.452827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:14.419424 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:14.419373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7599dbb675-654xh" event={"ID":"932a9a20-ca3f-4d8b-990c-f43a778f6b94","Type":"ContainerStarted","Data":"03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347"} Sep 15 20:42:14.436495 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:14.436448 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7599dbb675-654xh" podStartSLOduration=1.639570692 podStartE2EDuration="4.43643387s" totalImagesPullingTime="2.796863178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:42:14.435052072 +0000 UTC m=+323.564788654" watchObservedRunningTime="2026-09-15 20:42:14.43643387 +0000 UTC m=+323.566170447" Sep 15 20:42:14.502899 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:14.502871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:14.553812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:14.553781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:16.513639 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:16.513604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:19.717469 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:19.717400 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:19.771758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:19.771728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:20.823917 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:20.823876 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:20.824283 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:20.823945 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:20.829725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:20.829702 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:21.450427 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:21.450387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:42:22.504364 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:22.504328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:23.509970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:23.509934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:23.561770 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:23.561723 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:25.464875 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:25.464843 2580 generic.go:356] "Generic (PLEG): container finished" podID="9312d9bc-0d1b-48c8-8e90-ebe08397042f" containerID="1dd8b47922363a87c80ee70d89572fa0c43d05d000817c5a478aa67b69cad61e" exitCode=0 Sep 15 20:42:25.465265 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:25.464883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-27q29" event={"ID":"9312d9bc-0d1b-48c8-8e90-ebe08397042f","Type":"ContainerDied","Data":"1dd8b47922363a87c80ee70d89572fa0c43d05d000817c5a478aa67b69cad61e"} Sep 15 20:42:25.465265 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:25.465203 2580 scope.go:117] "RemoveContainer" containerID="1dd8b47922363a87c80ee70d89572fa0c43d05d000817c5a478aa67b69cad61e" Sep 15 20:42:26.470579 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:26.470543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-27q29" event={"ID":"9312d9bc-0d1b-48c8-8e90-ebe08397042f","Type":"ContainerStarted","Data":"7d1709e53318ef7253be50da653dc5fb2209e88f219e5448560607ec580437b2"} Sep 15 20:42:34.720932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:34.720895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:34.777998 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:34.777956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:59.704855 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:59.704815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:42:59.758271 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:42:59.758241 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:03.516421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:03.516375 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:03.829253 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:03.829160 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:04.580001 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:04.579966 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:43:04.580483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:04.580339 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="alertmanager" containerID="cri-o://486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f" gracePeriod=120 Sep 15 20:43:04.580483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:04.580415 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-web" containerID="cri-o://320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0" gracePeriod=120 Sep 15 20:43:04.580483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:04.580439 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="prom-label-proxy" containerID="cri-o://997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f" gracePeriod=120 Sep 15 20:43:04.580483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:04.580449 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy" containerID="cri-o://ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97" gracePeriod=120 Sep 15 20:43:04.580692 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:04.580397 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-metric" containerID="cri-o://8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f" gracePeriod=120 Sep 15 20:43:04.580692 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:04.580462 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="config-reloader" containerID="cri-o://26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099" gracePeriod=120 Sep 15 20:43:05.176395 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.176361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:43:05.178619 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.178587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad0237e-3438-4e30-a3f8-5f76d25d0763-metrics-certs\") pod \"network-metrics-daemon-5vvbb\" (UID: \"dad0237e-3438-4e30-a3f8-5f76d25d0763\") " pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:43:05.411377 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.411345 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxplx\"" Sep 15 20:43:05.419147 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.419126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vvbb" Sep 15 20:43:05.631057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631025 2580 generic.go:356] "Generic (PLEG): container finished" podID="07f7977a-73ac-4881-b12f-05646ea233f7" containerID="997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f" exitCode=0 Sep 15 20:43:05.631057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631050 2580 generic.go:356] "Generic (PLEG): container finished" podID="07f7977a-73ac-4881-b12f-05646ea233f7" containerID="ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97" exitCode=0 Sep 15 20:43:05.631057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631057 2580 generic.go:356] "Generic (PLEG): container finished" podID="07f7977a-73ac-4881-b12f-05646ea233f7" containerID="26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099" exitCode=0 Sep 15 20:43:05.631057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631063 2580 generic.go:356] "Generic (PLEG): container finished" podID="07f7977a-73ac-4881-b12f-05646ea233f7" containerID="486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f" exitCode=0 Sep 15 20:43:05.631564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f"} Sep 15 20:43:05.631564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631186 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97"} Sep 15 20:43:05.631564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099"} Sep 15 20:43:05.631564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.631272 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f"} Sep 15 20:43:05.687278 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.687249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5vvbb"] Sep 15 20:43:05.710539 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:43:05.710505 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad0237e_3438_4e30_a3f8_5f76d25d0763.slice/crio-c461bc92f8a9bbfe6b357558e5d95f012418e804b746ec197d718dfa840974aa WatchSource:0}: Error finding container c461bc92f8a9bbfe6b357558e5d95f012418e804b746ec197d718dfa840974aa: Status 404 returned error can't find the container with id c461bc92f8a9bbfe6b357558e5d95f012418e804b746ec197d718dfa840974aa Sep 15 20:43:05.817370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.817348 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.882929 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.882983 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-web-config\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883012 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-main-db\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-tls-assets\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883066 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883099 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883142 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbv96\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-kube-api-access-pbv96\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883171 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-metrics-client-ca\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883193 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-config-out\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883223 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-main-tls\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-config-volume\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883297 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-cluster-tls-config\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.883322 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy\") pod \"07f7977a-73ac-4881-b12f-05646ea233f7\" (UID: \"07f7977a-73ac-4881-b12f-05646ea233f7\") " Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.886231 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.886696 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.891709 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:43:05.894426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.892567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:05.896673 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.896367 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:43:05.903117 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.903069 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:05.903370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.903331 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-kube-api-access-pbv96" (OuterVolumeSpecName: "kube-api-access-pbv96") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "kube-api-access-pbv96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:43:05.904320 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.904296 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:05.904470 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.904362 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-config-volume" (OuterVolumeSpecName: "config-volume") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:05.904542 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.904513 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-config-out" (OuterVolumeSpecName: "config-out") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:43:05.904604 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.904540 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:05.906056 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.906024 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:05.910287 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.910264 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-web-config" (OuterVolumeSpecName: "web-config") pod "07f7977a-73ac-4881-b12f-05646ea233f7" (UID: "07f7977a-73ac-4881-b12f-05646ea233f7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:05.984640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984551 2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-cluster-tls-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984580 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984597 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984611 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-web-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984626 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-alertmanager-main-db\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984643 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-tls-assets\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984655 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984663 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984676 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbv96\" (UniqueName: \"kubernetes.io/projected/07f7977a-73ac-4881-b12f-05646ea233f7-kube-api-access-pbv96\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984689 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f7977a-73ac-4881-b12f-05646ea233f7-metrics-client-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984698 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07f7977a-73ac-4881-b12f-05646ea233f7-config-out\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984706 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-secret-alertmanager-main-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:05.984945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:05.984714 2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07f7977a-73ac-4881-b12f-05646ea233f7-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:06.005134 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.005105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:06.059710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.059675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:06.640171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.640141 2580 generic.go:356] "Generic (PLEG): container finished" podID="07f7977a-73ac-4881-b12f-05646ea233f7" containerID="8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f" exitCode=0 Sep 15 20:43:06.640171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.640170 2580 generic.go:356] "Generic (PLEG): container finished" podID="07f7977a-73ac-4881-b12f-05646ea233f7" containerID="320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0" exitCode=0 Sep 15 20:43:06.640635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.640221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f"} Sep 15 20:43:06.640635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.640249 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.640635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.640320 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0"} Sep 15 20:43:06.640635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.640370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"07f7977a-73ac-4881-b12f-05646ea233f7","Type":"ContainerDied","Data":"53f89a783cc82a58bf6a69091badcc5c3c0da1aea46766b0480e0e38a15f95ee"} Sep 15 20:43:06.640635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.640395 2580 scope.go:117] "RemoveContainer" containerID="997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f" Sep 15 20:43:06.642174 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.642092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5vvbb" event={"ID":"dad0237e-3438-4e30-a3f8-5f76d25d0763","Type":"ContainerStarted","Data":"40dccef3b2a85a2b8328924d406d2bfa7e2a180aec61b913daff9a68f07dfea9"} Sep 15 20:43:06.642174 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.642140 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5vvbb" event={"ID":"dad0237e-3438-4e30-a3f8-5f76d25d0763","Type":"ContainerStarted","Data":"c461bc92f8a9bbfe6b357558e5d95f012418e804b746ec197d718dfa840974aa"} Sep 15 20:43:06.650177 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.650157 2580 scope.go:117] "RemoveContainer" containerID="8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f" Sep 15 20:43:06.665775 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.665750 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:43:06.670121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.670096 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:43:06.674567 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.674515 2580 scope.go:117] "RemoveContainer" containerID="ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97" Sep 15 20:43:06.690216 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.690131 2580 scope.go:117] "RemoveContainer" containerID="320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0" Sep 15 20:43:06.694845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.694824 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:43:06.695236 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695221 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="prom-label-proxy" Sep 15 20:43:06.695288 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695238 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="prom-label-proxy" Sep 15 20:43:06.695288 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695247 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy" Sep 15 20:43:06.695288 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695253 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy" Sep 15 20:43:06.695288 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695263 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="config-reloader" Sep 15 20:43:06.695288 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695269 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="config-reloader" Sep 15 20:43:06.695288 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695285 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="init-config-reloader" Sep 15 20:43:06.695288 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695290 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="init-config-reloader" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695296 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-web" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695301 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-web" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695308 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-metric" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695314 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-metric" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695320 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="alertmanager" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695325 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="alertmanager" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695375 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="prom-label-proxy" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695381 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695387 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-metric" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695394 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="alertmanager" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695400 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="kube-rbac-proxy-web" Sep 15 20:43:06.695812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.695424 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" containerName="config-reloader" Sep 15 20:43:06.698517 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.698494 2580 scope.go:117] "RemoveContainer" containerID="26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099" Sep 15 20:43:06.699467 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.699447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.701779 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.701758 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 15 20:43:06.702096 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.701908 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 15 20:43:06.702096 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.701860 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 15 20:43:06.702096 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.702004 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 15 20:43:06.702096 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.702097 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 15 20:43:06.702394 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.702377 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 15 20:43:06.702490 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.702422 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 15 20:43:06.702598 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.702577 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nbc8f\"" Sep 15 20:43:06.703263 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.703245 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 15 20:43:06.708562 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.708550 2580 scope.go:117] "RemoveContainer" containerID="486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f" Sep 15 20:43:06.708727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.708709 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 15 20:43:06.722377 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.721992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:43:06.722377 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.722262 2580 scope.go:117] "RemoveContainer" containerID="2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19" Sep 15 20:43:06.731987 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.731963 2580 scope.go:117] "RemoveContainer" containerID="997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f" Sep 15 20:43:06.732252 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:06.732234 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f\": container with ID starting with 997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f not found: ID does not exist" containerID="997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f" Sep 15 20:43:06.732305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.732259 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f"} err="failed to get container status \"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f\": rpc error: code = NotFound desc = could not find container \"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f\": container with ID starting with 997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f not found: ID does not exist" Sep 15 20:43:06.732305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.732276 2580 scope.go:117] "RemoveContainer" containerID="8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f" Sep 15 20:43:06.732565 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:06.732546 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f\": container with ID starting with 8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f not found: ID does not exist" containerID="8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f" Sep 15 20:43:06.732613 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.732570 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f"} err="failed to get container status \"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f\": rpc error: code = NotFound desc = could not find container \"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f\": container with ID starting with 8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f not found: ID does not exist" Sep 15 20:43:06.732613 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.732584 2580 scope.go:117] "RemoveContainer" containerID="ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97" Sep 15 20:43:06.732811 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:06.732790 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97\": container with ID starting with ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97 not found: ID does not exist" containerID="ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97" Sep 15 20:43:06.732857 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.732817 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97"} err="failed to get container status \"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97\": rpc error: code = NotFound desc = could not find container \"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97\": container with ID starting with ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97 not found: ID does not exist" Sep 15 20:43:06.732857 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.732833 2580 scope.go:117] "RemoveContainer" containerID="320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0" Sep 15 20:43:06.733058 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:06.733039 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0\": container with ID starting with 320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0 not found: ID does not exist" containerID="320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0" Sep 15 20:43:06.733100 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733061 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0"} err="failed to get container status \"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0\": rpc error: code = NotFound desc = could not find container \"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0\": container with ID starting with 320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0 not found: ID does not exist" Sep 15 20:43:06.733100 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733072 2580 scope.go:117] "RemoveContainer" containerID="26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099" Sep 15 20:43:06.733292 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:06.733274 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099\": container with ID starting with 26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099 not found: ID does not exist" containerID="26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099" Sep 15 20:43:06.733333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733296 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099"} err="failed to get container status \"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099\": rpc error: code = NotFound desc = could not find container \"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099\": container with ID starting with 26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099 not found: ID does not exist" Sep 15 20:43:06.733333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733309 2580 scope.go:117] "RemoveContainer" containerID="486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f" Sep 15 20:43:06.733546 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:06.733529 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f\": container with ID starting with 486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f not found: ID does not exist" containerID="486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f" Sep 15 20:43:06.733594 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733550 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f"} err="failed to get container status \"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f\": rpc error: code = NotFound desc = could not find container \"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f\": container with ID starting with 486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f not found: ID does not exist" Sep 15 20:43:06.733594 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733563 2580 scope.go:117] "RemoveContainer" containerID="2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19" Sep 15 20:43:06.733784 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:06.733768 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19\": container with ID starting with 2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19 not found: ID does not exist" containerID="2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19" Sep 15 20:43:06.733824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733787 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19"} err="failed to get container status \"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19\": rpc error: code = NotFound desc = could not find container \"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19\": container with ID starting with 2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19 not found: ID does not exist" Sep 15 20:43:06.733824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733799 2580 scope.go:117] "RemoveContainer" containerID="997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f" Sep 15 20:43:06.734002 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.733985 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f"} err="failed to get container status \"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f\": rpc error: code = NotFound desc = could not find container \"997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f\": container with ID starting with 997eadf75ae492c2b520ea40eef1a53e1f81245da9ea885a12a43bad9643947f not found: ID does not exist" Sep 15 20:43:06.734048 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.734001 2580 scope.go:117] "RemoveContainer" containerID="8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f" Sep 15 20:43:06.734242 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.734220 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f"} err="failed to get container status \"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f\": rpc error: code = NotFound desc = could not find container \"8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f\": container with ID starting with 8f58fd3d44368ee6ad5e3394b98fc4539ba7e60653a9529cba558ec02af9ea1f not found: ID does not exist" Sep 15 20:43:06.734314 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.734244 2580 scope.go:117] "RemoveContainer" containerID="ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97" Sep 15 20:43:06.734547 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.734512 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97"} err="failed to get container status \"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97\": rpc error: code = NotFound desc = could not find container \"ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97\": container with ID starting with ee36772982b5f7acd9b37dcd4e803e812d7718b95efa9353edf1f6d5da684d97 not found: ID does not exist" Sep 15 20:43:06.734612 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.734548 2580 scope.go:117] "RemoveContainer" containerID="320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0" Sep 15 20:43:06.734825 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.734803 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0"} err="failed to get container status \"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0\": rpc error: code = NotFound desc = could not find container \"320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0\": container with ID starting with 320fb0f7684f43e5a8f157ec344f3f1810ddce2954007e911da6c93617081af0 not found: ID does not exist" Sep 15 20:43:06.734825 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.734824 2580 scope.go:117] "RemoveContainer" containerID="26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099" Sep 15 20:43:06.735112 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.735089 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099"} err="failed to get container status \"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099\": rpc error: code = NotFound desc = could not find container \"26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099\": container with ID starting with 26ce317316b05454448df8ff16281ed70d6b79bbd3478e0af98efb13e1e51099 not found: ID does not exist" Sep 15 20:43:06.735112 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.735113 2580 scope.go:117] "RemoveContainer" containerID="486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f" Sep 15 20:43:06.735732 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.735546 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f"} err="failed to get container status \"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f\": rpc error: code = NotFound desc = could not find container \"486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f\": container with ID starting with 486c2990fb2f776c9715d0e470b40d5ebb9e9a7b969f89d6fdfeacce596a156f not found: ID does not exist" Sep 15 20:43:06.735732 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.735715 2580 scope.go:117] "RemoveContainer" containerID="2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19" Sep 15 20:43:06.736043 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.736023 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19"} err="failed to get container status \"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19\": rpc error: code = NotFound desc = could not find container \"2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19\": container with ID starting with 2c04129c41fc4d836d0448995b74cf11abfa4f1a796ced060e85a2e6ff578c19 not found: ID does not exist" Sep 15 20:43:06.793446 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793446 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03425d96-de2e-4bb4-9c24-67e304014b62-config-out\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx29s\" (UniqueName: \"kubernetes.io/projected/03425d96-de2e-4bb4-9c24-67e304014b62-kube-api-access-qx29s\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793650 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03425d96-de2e-4bb4-9c24-67e304014b62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793650 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-config-volume\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03425d96-de2e-4bb4-9c24-67e304014b62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793768 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03425d96-de2e-4bb4-9c24-67e304014b62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793768 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793753 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03425d96-de2e-4bb4-9c24-67e304014b62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793829 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-web-config\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.793829 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.793793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.894779 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.894749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03425d96-de2e-4bb4-9c24-67e304014b62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.894779 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.894783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03425d96-de2e-4bb4-9c24-67e304014b62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.894909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.894802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03425d96-de2e-4bb4-9c24-67e304014b62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.894909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.894817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-web-config\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895011 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.894955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895011 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895116 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03425d96-de2e-4bb4-9c24-67e304014b62-config-out\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895116 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx29s\" (UniqueName: \"kubernetes.io/projected/03425d96-de2e-4bb4-9c24-67e304014b62-kube-api-access-qx29s\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895116 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03425d96-de2e-4bb4-9c24-67e304014b62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-config-volume\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895484 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03425d96-de2e-4bb4-9c24-67e304014b62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.895587 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.895565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03425d96-de2e-4bb4-9c24-67e304014b62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.896658 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.896335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03425d96-de2e-4bb4-9c24-67e304014b62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.897852 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.897829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03425d96-de2e-4bb4-9c24-67e304014b62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.897950 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.897835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03425d96-de2e-4bb4-9c24-67e304014b62-config-out\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.898009 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.897968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.898101 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.898077 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-config-volume\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.898326 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.898299 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.898448 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.898368 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.898589 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.898570 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-web-config\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.899017 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.898999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.899798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.899777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03425d96-de2e-4bb4-9c24-67e304014b62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:06.903559 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:06.903539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx29s\" (UniqueName: \"kubernetes.io/projected/03425d96-de2e-4bb4-9c24-67e304014b62-kube-api-access-qx29s\") pod \"alertmanager-main-0\" (UID: \"03425d96-de2e-4bb4-9c24-67e304014b62\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:07.012655 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.012616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 20:43:07.292203 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.291752 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 20:43:07.296005 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:43:07.295975 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03425d96_de2e_4bb4_9c24_67e304014b62.slice/crio-c5a8b270d19182d60dc123de9cd414e2837e4cae0ff62b9e41c7aed49a4f1bfc WatchSource:0}: Error finding container c5a8b270d19182d60dc123de9cd414e2837e4cae0ff62b9e41c7aed49a4f1bfc: Status 404 returned error can't find the container with id c5a8b270d19182d60dc123de9cd414e2837e4cae0ff62b9e41c7aed49a4f1bfc Sep 15 20:43:07.312360 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.312298 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f7977a-73ac-4881-b12f-05646ea233f7" path="/var/lib/kubelet/pods/07f7977a-73ac-4881-b12f-05646ea233f7/volumes" Sep 15 20:43:07.647543 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.647456 2580 generic.go:356] "Generic (PLEG): container finished" podID="03425d96-de2e-4bb4-9c24-67e304014b62" containerID="16f8bd0556a9e6f3e121f2d070e1355c7cf725dc413de0e429bed55187230495" exitCode=0 Sep 15 20:43:07.647963 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.647554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerDied","Data":"16f8bd0556a9e6f3e121f2d070e1355c7cf725dc413de0e429bed55187230495"} Sep 15 20:43:07.647963 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.647631 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerStarted","Data":"c5a8b270d19182d60dc123de9cd414e2837e4cae0ff62b9e41c7aed49a4f1bfc"} Sep 15 20:43:07.650395 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.650372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5vvbb" event={"ID":"dad0237e-3438-4e30-a3f8-5f76d25d0763","Type":"ContainerStarted","Data":"b862b62f6e386e7d2da8fec6c60ee1fa16dea8e8f9ca2ccaeb1903c3bcc2fcf2"} Sep 15 20:43:07.685993 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.685949 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5vvbb" podStartSLOduration=374.879862575 podStartE2EDuration="6m15.685930105s" totalImagesPullingTime="806.06753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:43:07.684988108 +0000 UTC m=+376.814724690" watchObservedRunningTime="2026-09-15 20:43:07.685930105 +0000 UTC m=+376.815666688" Sep 15 20:43:07.769690 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.769664 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:07.872017 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:07.871994 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:08.597580 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.597546 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-66bb65b95-rdm5m"] Sep 15 20:43:08.601313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.601262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.603335 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.603307 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 15 20:43:08.603483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.603427 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 15 20:43:08.603483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.603430 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 15 20:43:08.603588 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.603508 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7pn6z\"" Sep 15 20:43:08.603636 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.603587 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 15 20:43:08.603845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.603825 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 15 20:43:08.608359 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.608335 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 15 20:43:08.613593 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.613570 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66bb65b95-rdm5m"] Sep 15 20:43:08.657388 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.657351 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerStarted","Data":"815c33b2cfac477f4ea8c43038941d0cd9cd0f7fe4173222203b8f25ede5a3fd"} Sep 15 20:43:08.657823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.657439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerStarted","Data":"b8a74718ff8300db5ab0052245a918ddd0fec28e10f78ddc88134259e8873a09"} Sep 15 20:43:08.657823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.657497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerStarted","Data":"4d130bd1d037d1093b7db06448fd627065b1aaa6fac985da20ae3e8f5953e798"} Sep 15 20:43:08.657823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.657545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerStarted","Data":"a725193953cb453255233037021c67e787d20a7779b97f32463adbde85685a5f"} Sep 15 20:43:08.657823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.657586 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerStarted","Data":"4320fa0c43b39144c22728533db2c7dd8522f02cee141616ce3bfd73c8da6a25"} Sep 15 20:43:08.657823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.657625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03425d96-de2e-4bb4-9c24-67e304014b62","Type":"ContainerStarted","Data":"b81538c484aa7c01520eb73744d319dc3ab54ff0b92bc1a29d64f567c13470c0"} Sep 15 20:43:08.682925 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.682874 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.664066881 podStartE2EDuration="2.682859078s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.792197ms" isStatefulPod=false podCreationTimestamp="2026-09-15 20:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:43:08.681100626 +0000 UTC m=+377.810837249" watchObservedRunningTime="2026-09-15 20:43:08.682859078 +0000 UTC m=+377.812595661" Sep 15 20:43:08.709981 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.709946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th6df\" (UniqueName: \"kubernetes.io/projected/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-kube-api-access-th6df\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.709981 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.709980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.710155 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.710014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-serving-certs-ca-bundle\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.710155 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.710031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-secret-telemeter-client\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.710155 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.710078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.710155 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.710100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-telemeter-client-tls\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.710155 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.710148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-metrics-client-ca\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.710313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.710179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-federate-client-tls\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.810968 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.810940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-metrics-client-ca\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.810988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-federate-client-tls\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.811030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th6df\" (UniqueName: \"kubernetes.io/projected/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-kube-api-access-th6df\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.811058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.811185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-serving-certs-ca-bundle\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.811214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-secret-telemeter-client\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.811314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.811391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-telemeter-client-tls\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.811811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.811783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-metrics-client-ca\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.812132 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.812105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.812319 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.812295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-serving-certs-ca-bundle\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.818858 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.814696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.819519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.819219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-telemeter-client-tls\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.819519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.819308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-secret-telemeter-client\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.819519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.819392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-federate-client-tls\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.820453 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.820431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th6df\" (UniqueName: \"kubernetes.io/projected/c6e0c9bd-9ee5-4226-ae50-3a571fb52d69-kube-api-access-th6df\") pod \"telemeter-client-66bb65b95-rdm5m\" (UID: \"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69\") " pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:08.911251 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:08.911198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" Sep 15 20:43:09.188234 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:09.188134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66bb65b95-rdm5m"] Sep 15 20:43:09.190381 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:43:09.190357 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e0c9bd_9ee5_4226_ae50_3a571fb52d69.slice/crio-b7949d2fb987f70029936f6bddea7d40d6bd4137b48f628b1b3d848230d53a4c WatchSource:0}: Error finding container b7949d2fb987f70029936f6bddea7d40d6bd4137b48f628b1b3d848230d53a4c: Status 404 returned error can't find the container with id b7949d2fb987f70029936f6bddea7d40d6bd4137b48f628b1b3d848230d53a4c Sep 15 20:43:09.662729 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:09.662689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" event={"ID":"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69","Type":"ContainerStarted","Data":"b7949d2fb987f70029936f6bddea7d40d6bd4137b48f628b1b3d848230d53a4c"} Sep 15 20:43:11.673626 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.673589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" event={"ID":"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69","Type":"ContainerStarted","Data":"012b5a3a7256d32d3622f3204aa67f248940d190124c5a81fc61808802a6c77d"} Sep 15 20:43:11.673999 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.673652 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" event={"ID":"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69","Type":"ContainerStarted","Data":"78ccd7c445601abea3664231e8c8ea91aa4d8b98b9d765d6edfe66b19005b307"} Sep 15 20:43:11.673999 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.673678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" event={"ID":"c6e0c9bd-9ee5-4226-ae50-3a571fb52d69","Type":"ContainerStarted","Data":"b352828533824b8d4ec6852225686f20464a7d43a961e861f6245a8d0f4bd53d"} Sep 15 20:43:11.693330 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.693288 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-66bb65b95-rdm5m" podStartSLOduration=1.794531706 podStartE2EDuration="3.693272055s" totalImagesPullingTime="1.898740349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:43:11.693056452 +0000 UTC m=+380.822793034" watchObservedRunningTime="2026-09-15 20:43:11.693272055 +0000 UTC m=+380.823008639" Sep 15 20:43:11.767291 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.767260 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64cbfd9f5-9d9ml"] Sep 15 20:43:11.770889 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.770866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:11.782226 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.782201 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cbfd9f5-9d9ml"] Sep 15 20:43:11.808357 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.808328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:11.859758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.859734 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:11.935630 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.935567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-service-ca\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:11.935630 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.935608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-oauth-serving-cert\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:11.935792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.935639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-serving-cert\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:11.935792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.935671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-console-config\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:11.935792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.935712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-trusted-ca-bundle\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:11.935792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.935759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnldg\" (UniqueName: \"kubernetes.io/projected/b6011ecb-91c2-4004-966b-b138cf01d15f-kube-api-access-mnldg\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:11.935949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:11.935822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-oauth-config\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.036976 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.036950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-service-ca\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037095 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.036990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-oauth-serving-cert\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037095 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-serving-cert\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037095 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-console-config\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037244 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-trusted-ca-bundle\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037244 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnldg\" (UniqueName: \"kubernetes.io/projected/b6011ecb-91c2-4004-966b-b138cf01d15f-kube-api-access-mnldg\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037399 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-oauth-config\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-console-config\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037883 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-oauth-serving-cert\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.037940 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.037904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-service-ca\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.038050 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.038034 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-trusted-ca-bundle\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.039558 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.039538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-serving-cert\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.039713 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.039698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-oauth-config\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.044945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.044925 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnldg\" (UniqueName: \"kubernetes.io/projected/b6011ecb-91c2-4004-966b-b138cf01d15f-kube-api-access-mnldg\") pod \"console-64cbfd9f5-9d9ml\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.082859 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.082837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:12.354049 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.354022 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cbfd9f5-9d9ml"] Sep 15 20:43:12.356250 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:43:12.356225 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6011ecb_91c2_4004_966b_b138cf01d15f.slice/crio-320d48876f050bc02a08022ce0a734845d9d8b196d0ac6f37ec459720ccf5150 WatchSource:0}: Error finding container 320d48876f050bc02a08022ce0a734845d9d8b196d0ac6f37ec459720ccf5150: Status 404 returned error can't find the container with id 320d48876f050bc02a08022ce0a734845d9d8b196d0ac6f37ec459720ccf5150 Sep 15 20:43:12.679335 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.679254 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cbfd9f5-9d9ml" event={"ID":"b6011ecb-91c2-4004-966b-b138cf01d15f","Type":"ContainerStarted","Data":"f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d"} Sep 15 20:43:12.679335 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.679305 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cbfd9f5-9d9ml" event={"ID":"b6011ecb-91c2-4004-966b-b138cf01d15f","Type":"ContainerStarted","Data":"320d48876f050bc02a08022ce0a734845d9d8b196d0ac6f37ec459720ccf5150"} Sep 15 20:43:12.695737 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:12.695689 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64cbfd9f5-9d9ml" podStartSLOduration=1.695674514 podStartE2EDuration="1.695674514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:43:12.695261833 +0000 UTC m=+381.824998416" watchObservedRunningTime="2026-09-15 20:43:12.695674514 +0000 UTC m=+381.825411097" Sep 15 20:43:22.083751 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.083707 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:22.084119 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.083767 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:22.088171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.088147 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:22.732207 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.732174 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:43:22.778324 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.778281 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7599dbb675-654xh"] Sep 15 20:43:22.879040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.878979 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:22.929338 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.929305 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:22.982007 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:22.981980 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:32.703873 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:32.703841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:47.808142 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:47.808098 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7599dbb675-654xh" podUID="932a9a20-ca3f-4d8b-990c-f43a778f6b94" containerName="console" containerID="cri-o://03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347" gracePeriod=15 Sep 15 20:43:48.045564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.045541 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:43:48.120955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.120888 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsl8m\" (UniqueName: \"kubernetes.io/projected/932a9a20-ca3f-4d8b-990c-f43a778f6b94-kube-api-access-bsl8m\") pod \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " Sep 15 20:43:48.120955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.120931 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-oauth-config\") pod \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " Sep 15 20:43:48.121161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.120963 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-serving-cert\") pod \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " Sep 15 20:43:48.121161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.120979 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-oauth-serving-cert\") pod \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " Sep 15 20:43:48.121161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121002 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-service-ca\") pod \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " Sep 15 20:43:48.121161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121142 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-trusted-ca-bundle\") pod \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " Sep 15 20:43:48.121363 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121183 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-config\") pod \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\" (UID: \"932a9a20-ca3f-4d8b-990c-f43a778f6b94\") " Sep 15 20:43:48.121530 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121480 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-service-ca" (OuterVolumeSpecName: "service-ca") pod "932a9a20-ca3f-4d8b-990c-f43a778f6b94" (UID: "932a9a20-ca3f-4d8b-990c-f43a778f6b94"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:48.121530 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121491 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "932a9a20-ca3f-4d8b-990c-f43a778f6b94" (UID: "932a9a20-ca3f-4d8b-990c-f43a778f6b94"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:48.121748 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121598 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:48.121748 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121657 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "932a9a20-ca3f-4d8b-990c-f43a778f6b94" (UID: "932a9a20-ca3f-4d8b-990c-f43a778f6b94"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:48.121817 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.121766 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-config" (OuterVolumeSpecName: "console-config") pod "932a9a20-ca3f-4d8b-990c-f43a778f6b94" (UID: "932a9a20-ca3f-4d8b-990c-f43a778f6b94"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:43:48.123261 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.123233 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "932a9a20-ca3f-4d8b-990c-f43a778f6b94" (UID: "932a9a20-ca3f-4d8b-990c-f43a778f6b94"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:48.123361 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.123288 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "932a9a20-ca3f-4d8b-990c-f43a778f6b94" (UID: "932a9a20-ca3f-4d8b-990c-f43a778f6b94"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:43:48.123361 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.123302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932a9a20-ca3f-4d8b-990c-f43a778f6b94-kube-api-access-bsl8m" (OuterVolumeSpecName: "kube-api-access-bsl8m") pod "932a9a20-ca3f-4d8b-990c-f43a778f6b94" (UID: "932a9a20-ca3f-4d8b-990c-f43a778f6b94"). InnerVolumeSpecName "kube-api-access-bsl8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:43:48.222108 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.222082 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsl8m\" (UniqueName: \"kubernetes.io/projected/932a9a20-ca3f-4d8b-990c-f43a778f6b94-kube-api-access-bsl8m\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:48.222108 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.222106 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-oauth-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:48.222264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.222116 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:48.222264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.222125 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-oauth-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:48.222264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.222133 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-trusted-ca-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:48.222264 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.222141 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/932a9a20-ca3f-4d8b-990c-f43a778f6b94-console-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:43:48.834715 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.834677 2580 generic.go:356] "Generic (PLEG): container finished" podID="932a9a20-ca3f-4d8b-990c-f43a778f6b94" containerID="03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347" exitCode=0 Sep 15 20:43:48.835145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.834727 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7599dbb675-654xh" Sep 15 20:43:48.835145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.834750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7599dbb675-654xh" event={"ID":"932a9a20-ca3f-4d8b-990c-f43a778f6b94","Type":"ContainerDied","Data":"03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347"} Sep 15 20:43:48.835145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.834793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7599dbb675-654xh" event={"ID":"932a9a20-ca3f-4d8b-990c-f43a778f6b94","Type":"ContainerDied","Data":"d0858c3ce3e1b059cfb1c41d950f018d94d4dc868cc3227b7a32c573bdb01a06"} Sep 15 20:43:48.835145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.834812 2580 scope.go:117] "RemoveContainer" containerID="03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347" Sep 15 20:43:48.843885 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.843866 2580 scope.go:117] "RemoveContainer" containerID="03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347" Sep 15 20:43:48.844166 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:43:48.844146 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347\": container with ID starting with 03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347 not found: ID does not exist" containerID="03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347" Sep 15 20:43:48.844210 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.844173 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347"} err="failed to get container status \"03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347\": rpc error: code = NotFound desc = could not find container \"03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347\": container with ID starting with 03a261170a4f5cc064cd46b9975ebebb473f062996cbbd20da6e05143c7bf347 not found: ID does not exist" Sep 15 20:43:48.855732 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.855712 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7599dbb675-654xh"] Sep 15 20:43:48.859379 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:48.859355 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7599dbb675-654xh"] Sep 15 20:43:49.318516 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:49.318475 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932a9a20-ca3f-4d8b-990c-f43a778f6b94" path="/var/lib/kubelet/pods/932a9a20-ca3f-4d8b-990c-f43a778f6b94/volumes" Sep 15 20:43:56.107175 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:56.107139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:43:56.161760 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:43:56.161730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:18.743767 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:18.743681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:18.802351 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:18.802297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:29.722008 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:29.721960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:30.671376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:30.671344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:30.721084 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:30.721059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:34.726342 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:34.726301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:34.824416 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:34.824388 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:35.002045 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:35.001992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:40.274955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:40.274921 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cbfd9f5-9d9ml"] Sep 15 20:44:40.394615 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:40.394582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:40.495387 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:40.495322 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:44:40.546630 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:44:40.546550 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:05.296163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.296121 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64cbfd9f5-9d9ml" podUID="b6011ecb-91c2-4004-966b-b138cf01d15f" containerName="console" containerID="cri-o://f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d" gracePeriod=15 Sep 15 20:45:05.533253 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.533230 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:45:05.597806 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.597743 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-oauth-config\") pod \"b6011ecb-91c2-4004-966b-b138cf01d15f\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " Sep 15 20:45:05.597806 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.597799 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-service-ca\") pod \"b6011ecb-91c2-4004-966b-b138cf01d15f\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " Sep 15 20:45:05.597997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.597820 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-oauth-serving-cert\") pod \"b6011ecb-91c2-4004-966b-b138cf01d15f\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " Sep 15 20:45:05.597997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.597838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-serving-cert\") pod \"b6011ecb-91c2-4004-966b-b138cf01d15f\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " Sep 15 20:45:05.597997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.597853 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-console-config\") pod \"b6011ecb-91c2-4004-966b-b138cf01d15f\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " Sep 15 20:45:05.597997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.597870 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnldg\" (UniqueName: \"kubernetes.io/projected/b6011ecb-91c2-4004-966b-b138cf01d15f-kube-api-access-mnldg\") pod \"b6011ecb-91c2-4004-966b-b138cf01d15f\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " Sep 15 20:45:05.597997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.597901 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-trusted-ca-bundle\") pod \"b6011ecb-91c2-4004-966b-b138cf01d15f\" (UID: \"b6011ecb-91c2-4004-966b-b138cf01d15f\") " Sep 15 20:45:05.598391 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.598276 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6011ecb-91c2-4004-966b-b138cf01d15f" (UID: "b6011ecb-91c2-4004-966b-b138cf01d15f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:45:05.598537 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.598270 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-console-config" (OuterVolumeSpecName: "console-config") pod "b6011ecb-91c2-4004-966b-b138cf01d15f" (UID: "b6011ecb-91c2-4004-966b-b138cf01d15f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:45:05.598537 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.598395 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6011ecb-91c2-4004-966b-b138cf01d15f" (UID: "b6011ecb-91c2-4004-966b-b138cf01d15f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:45:05.598537 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.598465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b6011ecb-91c2-4004-966b-b138cf01d15f" (UID: "b6011ecb-91c2-4004-966b-b138cf01d15f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:45:05.599907 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.599883 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6011ecb-91c2-4004-966b-b138cf01d15f" (UID: "b6011ecb-91c2-4004-966b-b138cf01d15f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:45:05.600355 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.600338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6011ecb-91c2-4004-966b-b138cf01d15f" (UID: "b6011ecb-91c2-4004-966b-b138cf01d15f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:45:05.600451 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.600422 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6011ecb-91c2-4004-966b-b138cf01d15f-kube-api-access-mnldg" (OuterVolumeSpecName: "kube-api-access-mnldg") pod "b6011ecb-91c2-4004-966b-b138cf01d15f" (UID: "b6011ecb-91c2-4004-966b-b138cf01d15f"). InnerVolumeSpecName "kube-api-access-mnldg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:45:05.699036 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.699013 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:05.699036 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.699034 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-oauth-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:05.699179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.699044 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:05.699179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.699054 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-console-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:05.699179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.699062 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnldg\" (UniqueName: \"kubernetes.io/projected/b6011ecb-91c2-4004-966b-b138cf01d15f-kube-api-access-mnldg\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:05.699179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.699070 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6011ecb-91c2-4004-966b-b138cf01d15f-trusted-ca-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:05.699179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:05.699079 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6011ecb-91c2-4004-966b-b138cf01d15f-console-oauth-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:06.147571 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.147533 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6011ecb-91c2-4004-966b-b138cf01d15f" containerID="f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d" exitCode=0 Sep 15 20:45:06.147742 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.147581 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cbfd9f5-9d9ml" Sep 15 20:45:06.147742 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.147581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cbfd9f5-9d9ml" event={"ID":"b6011ecb-91c2-4004-966b-b138cf01d15f","Type":"ContainerDied","Data":"f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d"} Sep 15 20:45:06.147742 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.147718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cbfd9f5-9d9ml" event={"ID":"b6011ecb-91c2-4004-966b-b138cf01d15f","Type":"ContainerDied","Data":"320d48876f050bc02a08022ce0a734845d9d8b196d0ac6f37ec459720ccf5150"} Sep 15 20:45:06.147890 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.147744 2580 scope.go:117] "RemoveContainer" containerID="f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d" Sep 15 20:45:06.156460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.156441 2580 scope.go:117] "RemoveContainer" containerID="f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d" Sep 15 20:45:06.156761 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:45:06.156737 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d\": container with ID starting with f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d not found: ID does not exist" containerID="f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d" Sep 15 20:45:06.156837 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.156769 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d"} err="failed to get container status \"f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d\": rpc error: code = NotFound desc = could not find container \"f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d\": container with ID starting with f3879c44ab94fe5f7929e850690328e15ef580e0fa9ffacbe2cacf5f5ae6a28d not found: ID does not exist" Sep 15 20:45:06.167219 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.167193 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cbfd9f5-9d9ml"] Sep 15 20:45:06.171049 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:06.171030 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64cbfd9f5-9d9ml"] Sep 15 20:45:07.313246 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:07.313209 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6011ecb-91c2-4004-966b-b138cf01d15f" path="/var/lib/kubelet/pods/b6011ecb-91c2-4004-966b-b138cf01d15f/volumes" Sep 15 20:45:21.321085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:21.321047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:21.425416 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:21.425380 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:22.033268 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.033235 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc"] Sep 15 20:45:22.033587 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.033574 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932a9a20-ca3f-4d8b-990c-f43a778f6b94" containerName="console" Sep 15 20:45:22.033635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.033588 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="932a9a20-ca3f-4d8b-990c-f43a778f6b94" containerName="console" Sep 15 20:45:22.033635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.033596 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6011ecb-91c2-4004-966b-b138cf01d15f" containerName="console" Sep 15 20:45:22.033635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.033601 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6011ecb-91c2-4004-966b-b138cf01d15f" containerName="console" Sep 15 20:45:22.033727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.033660 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="932a9a20-ca3f-4d8b-990c-f43a778f6b94" containerName="console" Sep 15 20:45:22.033727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.033670 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6011ecb-91c2-4004-966b-b138cf01d15f" containerName="console" Sep 15 20:45:22.036710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.036693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.039186 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.039161 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 20:45:22.039296 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.039161 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 20:45:22.039745 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.039724 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-745nq\"" Sep 15 20:45:22.044684 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.044662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc"] Sep 15 20:45:22.117948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.117909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.117948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.117951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.118175 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.118040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4hd\" (UniqueName: \"kubernetes.io/projected/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-kube-api-access-fv4hd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.218966 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.218926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.218966 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.218966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.219211 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.219034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4hd\" (UniqueName: \"kubernetes.io/projected/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-kube-api-access-fv4hd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.219305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.219282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.219356 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.219302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.227555 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.227524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4hd\" (UniqueName: \"kubernetes.io/projected/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-kube-api-access-fv4hd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.346043 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.345975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:22.616212 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:22.616189 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc"] Sep 15 20:45:22.618560 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:45:22.618534 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be859fb_10e3_40a1_bd3a_cfdeb5f9b16b.slice/crio-148fe22e2349472b85fea6d573320f6a651a08392bbd666148c4e82d25e1f610 WatchSource:0}: Error finding container 148fe22e2349472b85fea6d573320f6a651a08392bbd666148c4e82d25e1f610: Status 404 returned error can't find the container with id 148fe22e2349472b85fea6d573320f6a651a08392bbd666148c4e82d25e1f610 Sep 15 20:45:23.223845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:23.223808 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" event={"ID":"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b","Type":"ContainerStarted","Data":"148fe22e2349472b85fea6d573320f6a651a08392bbd666148c4e82d25e1f610"} Sep 15 20:45:26.242959 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:26.242925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:26.294904 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:26.294863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:32.265164 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:32.265131 2580 generic.go:356] "Generic (PLEG): container finished" podID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerID="2a25e0193493cc50e4265a42d9e996eaf0763c3bf9c1ee61756f3f9b33abcc86" exitCode=0 Sep 15 20:45:32.265585 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:32.265176 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" event={"ID":"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b","Type":"ContainerDied","Data":"2a25e0193493cc50e4265a42d9e996eaf0763c3bf9c1ee61756f3f9b33abcc86"} Sep 15 20:45:35.281482 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:35.281445 2580 generic.go:356] "Generic (PLEG): container finished" podID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerID="8cc48aa45b49c620424c5adc987cec3cb86743a90e1128032bc1d8b074eb8198" exitCode=0 Sep 15 20:45:35.281935 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:35.281524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" event={"ID":"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b","Type":"ContainerDied","Data":"8cc48aa45b49c620424c5adc987cec3cb86743a90e1128032bc1d8b074eb8198"} Sep 15 20:45:42.315493 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:42.315458 2580 generic.go:356] "Generic (PLEG): container finished" podID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerID="acbd746e699d43551040c9e9779edbc4683b9c1a5c85d2ca43cf37ba8c3decdd" exitCode=0 Sep 15 20:45:42.315896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:42.315547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" event={"ID":"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b","Type":"ContainerDied","Data":"acbd746e699d43551040c9e9779edbc4683b9c1a5c85d2ca43cf37ba8c3decdd"} Sep 15 20:45:43.486465 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.486435 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:43.604708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.604636 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-bundle\") pod \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " Sep 15 20:45:43.604708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.604676 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-util\") pod \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " Sep 15 20:45:43.604916 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.604725 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv4hd\" (UniqueName: \"kubernetes.io/projected/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-kube-api-access-fv4hd\") pod \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\" (UID: \"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b\") " Sep 15 20:45:43.605522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.605491 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-bundle" (OuterVolumeSpecName: "bundle") pod "4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" (UID: "4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:45:43.606789 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.606767 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-kube-api-access-fv4hd" (OuterVolumeSpecName: "kube-api-access-fv4hd") pod "4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" (UID: "4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b"). InnerVolumeSpecName "kube-api-access-fv4hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:45:43.610443 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.610395 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-util" (OuterVolumeSpecName: "util") pod "4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" (UID: "4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:45:43.705656 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.705625 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:43.705656 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.705653 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:43.705656 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:43.705662 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv4hd\" (UniqueName: \"kubernetes.io/projected/4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b-kube-api-access-fv4hd\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:45:44.325132 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:44.325103 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" Sep 15 20:45:44.325132 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:44.325116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402w2hc" event={"ID":"4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b","Type":"ContainerDied","Data":"148fe22e2349472b85fea6d573320f6a651a08392bbd666148c4e82d25e1f610"} Sep 15 20:45:44.325330 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:44.325161 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="148fe22e2349472b85fea6d573320f6a651a08392bbd666148c4e82d25e1f610" Sep 15 20:45:45.507093 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:45.507057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:48.053435 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:48.053382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:48.848050 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:48.848015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:51.662952 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:51.662913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:52.192253 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:52.192217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:53.857011 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:53.856975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:53.996198 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:53.996157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:54.093954 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.093920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:54.238903 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.238826 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rxhwd"] Sep 15 20:45:54.239146 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.239135 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerName="util" Sep 15 20:45:54.239194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.239148 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerName="util" Sep 15 20:45:54.239194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.239161 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerName="pull" Sep 15 20:45:54.239194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.239166 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerName="pull" Sep 15 20:45:54.239194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.239175 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerName="extract" Sep 15 20:45:54.239194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.239180 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerName="extract" Sep 15 20:45:54.239362 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.239234 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be859fb-10e3-40a1-bd3a-cfdeb5f9b16b" containerName="extract" Sep 15 20:45:54.243289 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.243270 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.245694 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.245675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 15 20:45:54.246398 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.246383 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 15 20:45:54.246516 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.246491 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gcxrs\"" Sep 15 20:45:54.255477 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.255454 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rxhwd"] Sep 15 20:45:54.332798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.332770 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq"] Sep 15 20:45:54.336188 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.336170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.338626 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.338601 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-745nq\"" Sep 15 20:45:54.338738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.338638 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 20:45:54.338738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.338654 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 20:45:54.344373 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.344354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq"] Sep 15 20:45:54.392854 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.392831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c9e9a5-5152-44cc-9870-904c16eddcc8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rxhwd\" (UID: \"d9c9e9a5-5152-44cc-9870-904c16eddcc8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.392951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.392893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7hr\" (UniqueName: \"kubernetes.io/projected/d9c9e9a5-5152-44cc-9870-904c16eddcc8-kube-api-access-tr7hr\") pod \"cert-manager-webhook-6486ffd997-rxhwd\" (UID: \"d9c9e9a5-5152-44cc-9870-904c16eddcc8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.493801 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.493741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.493801 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.493773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c9e9a5-5152-44cc-9870-904c16eddcc8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rxhwd\" (UID: \"d9c9e9a5-5152-44cc-9870-904c16eddcc8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.493928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.493861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7hr\" (UniqueName: \"kubernetes.io/projected/d9c9e9a5-5152-44cc-9870-904c16eddcc8-kube-api-access-tr7hr\") pod \"cert-manager-webhook-6486ffd997-rxhwd\" (UID: \"d9c9e9a5-5152-44cc-9870-904c16eddcc8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.493928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.493895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.493928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.493924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wjz\" (UniqueName: \"kubernetes.io/projected/f484cb1c-4129-4179-b714-d4aadb8da91e-kube-api-access-s7wjz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.501500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.501473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c9e9a5-5152-44cc-9870-904c16eddcc8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rxhwd\" (UID: \"d9c9e9a5-5152-44cc-9870-904c16eddcc8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.501586 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.501573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7hr\" (UniqueName: \"kubernetes.io/projected/d9c9e9a5-5152-44cc-9870-904c16eddcc8-kube-api-access-tr7hr\") pod \"cert-manager-webhook-6486ffd997-rxhwd\" (UID: \"d9c9e9a5-5152-44cc-9870-904c16eddcc8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.552312 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.552288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:54.594861 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.594823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.595012 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.594871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wjz\" (UniqueName: \"kubernetes.io/projected/f484cb1c-4129-4179-b714-d4aadb8da91e-kube-api-access-s7wjz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.595012 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.594912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.595246 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.595225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.595305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.595283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.603927 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.603899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wjz\" (UniqueName: \"kubernetes.io/projected/f484cb1c-4129-4179-b714-d4aadb8da91e-kube-api-access-s7wjz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.645969 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.645945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:45:54.825862 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.825835 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rxhwd"] Sep 15 20:45:54.827883 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:45:54.827859 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c9e9a5_5152_44cc_9870_904c16eddcc8.slice/crio-60dd970538fd52cbcbb8c992eff17f2ec0ca5bdf827d27a2a7fabd328af8f3a2 WatchSource:0}: Error finding container 60dd970538fd52cbcbb8c992eff17f2ec0ca5bdf827d27a2a7fabd328af8f3a2: Status 404 returned error can't find the container with id 60dd970538fd52cbcbb8c992eff17f2ec0ca5bdf827d27a2a7fabd328af8f3a2 Sep 15 20:45:54.921321 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:54.921251 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq"] Sep 15 20:45:54.923314 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:45:54.923289 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf484cb1c_4129_4179_b714_d4aadb8da91e.slice/crio-54497ae2cce8853889b54554d168921b8fb64cb33e20dda974199c0669da0e27 WatchSource:0}: Error finding container 54497ae2cce8853889b54554d168921b8fb64cb33e20dda974199c0669da0e27: Status 404 returned error can't find the container with id 54497ae2cce8853889b54554d168921b8fb64cb33e20dda974199c0669da0e27 Sep 15 20:45:55.369637 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:55.369598 2580 generic.go:356] "Generic (PLEG): container finished" podID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerID="9b04809c56a24a3208d9e52187b6416248c0adc09a7fedad31e9191fad190f94" exitCode=0 Sep 15 20:45:55.369812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:55.369680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" event={"ID":"f484cb1c-4129-4179-b714-d4aadb8da91e","Type":"ContainerDied","Data":"9b04809c56a24a3208d9e52187b6416248c0adc09a7fedad31e9191fad190f94"} Sep 15 20:45:55.369812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:55.369732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" event={"ID":"f484cb1c-4129-4179-b714-d4aadb8da91e","Type":"ContainerStarted","Data":"54497ae2cce8853889b54554d168921b8fb64cb33e20dda974199c0669da0e27"} Sep 15 20:45:55.370976 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:55.370952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" event={"ID":"d9c9e9a5-5152-44cc-9870-904c16eddcc8","Type":"ContainerStarted","Data":"60dd970538fd52cbcbb8c992eff17f2ec0ca5bdf827d27a2a7fabd328af8f3a2"} Sep 15 20:45:56.237083 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:56.237044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:56.659562 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:56.659531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:58.457033 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:58.456994 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:58.557807 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:58.557777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:45:59.390728 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:59.390692 2580 generic.go:356] "Generic (PLEG): container finished" podID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerID="2b5a0ec475b468063526347af93ae4e45be7459d09f6b8bb2d3236f0a781f432" exitCode=0 Sep 15 20:45:59.390924 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:59.390786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" event={"ID":"f484cb1c-4129-4179-b714-d4aadb8da91e","Type":"ContainerDied","Data":"2b5a0ec475b468063526347af93ae4e45be7459d09f6b8bb2d3236f0a781f432"} Sep 15 20:45:59.395426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:59.395380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" event={"ID":"d9c9e9a5-5152-44cc-9870-904c16eddcc8","Type":"ContainerStarted","Data":"58d782aba8625f1557aab492562f46bb5872a82daf905972c2aa68b58587b37b"} Sep 15 20:45:59.398078 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:59.398060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:45:59.425893 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:45:59.425845 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" podStartSLOduration=1.889844991 podStartE2EDuration="5.42582573s" totalImagesPullingTime="3.535980739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:45:59.425185654 +0000 UTC m=+548.554922237" watchObservedRunningTime="2026-09-15 20:45:59.42582573 +0000 UTC m=+548.555562314" Sep 15 20:46:00.401571 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:00.401536 2580 generic.go:356] "Generic (PLEG): container finished" podID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerID="845655e7c99b426db1ead11c97b3964ad9bd0ee35083f3dfd9e0121f62c472d1" exitCode=0 Sep 15 20:46:00.401955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:00.401611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" event={"ID":"f484cb1c-4129-4179-b714-d4aadb8da91e","Type":"ContainerDied","Data":"845655e7c99b426db1ead11c97b3964ad9bd0ee35083f3dfd9e0121f62c472d1"} Sep 15 20:46:01.538090 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.538066 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:46:01.655443 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.655351 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-util\") pod \"f484cb1c-4129-4179-b714-d4aadb8da91e\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " Sep 15 20:46:01.655443 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.655392 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7wjz\" (UniqueName: \"kubernetes.io/projected/f484cb1c-4129-4179-b714-d4aadb8da91e-kube-api-access-s7wjz\") pod \"f484cb1c-4129-4179-b714-d4aadb8da91e\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " Sep 15 20:46:01.655625 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.655496 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-bundle\") pod \"f484cb1c-4129-4179-b714-d4aadb8da91e\" (UID: \"f484cb1c-4129-4179-b714-d4aadb8da91e\") " Sep 15 20:46:01.655897 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.655862 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-bundle" (OuterVolumeSpecName: "bundle") pod "f484cb1c-4129-4179-b714-d4aadb8da91e" (UID: "f484cb1c-4129-4179-b714-d4aadb8da91e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:46:01.657470 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.657442 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f484cb1c-4129-4179-b714-d4aadb8da91e-kube-api-access-s7wjz" (OuterVolumeSpecName: "kube-api-access-s7wjz") pod "f484cb1c-4129-4179-b714-d4aadb8da91e" (UID: "f484cb1c-4129-4179-b714-d4aadb8da91e"). InnerVolumeSpecName "kube-api-access-s7wjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:46:01.659994 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.659967 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-util" (OuterVolumeSpecName: "util") pod "f484cb1c-4129-4179-b714-d4aadb8da91e" (UID: "f484cb1c-4129-4179-b714-d4aadb8da91e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:46:01.756541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.756508 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:01.756541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.756530 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f484cb1c-4129-4179-b714-d4aadb8da91e-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:01.756541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:01.756539 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wjz\" (UniqueName: \"kubernetes.io/projected/f484cb1c-4129-4179-b714-d4aadb8da91e-kube-api-access-s7wjz\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:02.410307 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:02.410274 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" Sep 15 20:46:02.410501 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:02.410299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7gq" event={"ID":"f484cb1c-4129-4179-b714-d4aadb8da91e","Type":"ContainerDied","Data":"54497ae2cce8853889b54554d168921b8fb64cb33e20dda974199c0669da0e27"} Sep 15 20:46:02.410501 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:02.410354 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54497ae2cce8853889b54554d168921b8fb64cb33e20dda974199c0669da0e27" Sep 15 20:46:03.460692 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:03.460654 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:04.048125 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:04.048090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:05.403730 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:05.403699 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rxhwd" Sep 15 20:46:06.880645 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.880612 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gcblf"] Sep 15 20:46:06.881000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.880920 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerName="util" Sep 15 20:46:06.881000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.880931 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerName="util" Sep 15 20:46:06.881000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.880947 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerName="pull" Sep 15 20:46:06.881000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.880952 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerName="pull" Sep 15 20:46:06.881000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.880958 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerName="extract" Sep 15 20:46:06.881000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.880964 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerName="extract" Sep 15 20:46:06.881192 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.881012 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f484cb1c-4129-4179-b714-d4aadb8da91e" containerName="extract" Sep 15 20:46:06.884217 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.884202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:06.886205 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.886183 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c2mcc\"" Sep 15 20:46:06.887841 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.887821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b313847-74b6-405e-89ae-0475316c9733-bound-sa-token\") pod \"cert-manager-55b4d954d5-gcblf\" (UID: \"5b313847-74b6-405e-89ae-0475316c9733\") " pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:06.887938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.887885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85gb\" (UniqueName: \"kubernetes.io/projected/5b313847-74b6-405e-89ae-0475316c9733-kube-api-access-g85gb\") pod \"cert-manager-55b4d954d5-gcblf\" (UID: \"5b313847-74b6-405e-89ae-0475316c9733\") " pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:06.891571 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.891548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gcblf"] Sep 15 20:46:06.988740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.988710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85gb\" (UniqueName: \"kubernetes.io/projected/5b313847-74b6-405e-89ae-0475316c9733-kube-api-access-g85gb\") pod \"cert-manager-55b4d954d5-gcblf\" (UID: \"5b313847-74b6-405e-89ae-0475316c9733\") " pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:06.988840 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.988786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b313847-74b6-405e-89ae-0475316c9733-bound-sa-token\") pod \"cert-manager-55b4d954d5-gcblf\" (UID: \"5b313847-74b6-405e-89ae-0475316c9733\") " pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:06.998010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.997984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b313847-74b6-405e-89ae-0475316c9733-bound-sa-token\") pod \"cert-manager-55b4d954d5-gcblf\" (UID: \"5b313847-74b6-405e-89ae-0475316c9733\") " pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:06.998284 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:06.998263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85gb\" (UniqueName: \"kubernetes.io/projected/5b313847-74b6-405e-89ae-0475316c9733-kube-api-access-g85gb\") pod \"cert-manager-55b4d954d5-gcblf\" (UID: \"5b313847-74b6-405e-89ae-0475316c9733\") " pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:07.195351 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:07.195252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gcblf" Sep 15 20:46:07.463190 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:07.463116 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gcblf"] Sep 15 20:46:07.467221 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:46:07.467191 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b313847_74b6_405e_89ae_0475316c9733.slice/crio-594973b89032a0baec3fccd230d34f3797e91c511794e39a4ba3e9b3fa5f0ac8 WatchSource:0}: Error finding container 594973b89032a0baec3fccd230d34f3797e91c511794e39a4ba3e9b3fa5f0ac8: Status 404 returned error can't find the container with id 594973b89032a0baec3fccd230d34f3797e91c511794e39a4ba3e9b3fa5f0ac8 Sep 15 20:46:08.437723 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:08.437689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gcblf" event={"ID":"5b313847-74b6-405e-89ae-0475316c9733","Type":"ContainerStarted","Data":"97783ab919e733c5cad832b7d85e7a9a259f52a2c3137b6e23bdc49bf428124a"} Sep 15 20:46:08.438146 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:08.437731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gcblf" event={"ID":"5b313847-74b6-405e-89ae-0475316c9733","Type":"ContainerStarted","Data":"594973b89032a0baec3fccd230d34f3797e91c511794e39a4ba3e9b3fa5f0ac8"} Sep 15 20:46:08.453067 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:08.453023 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gcblf" podStartSLOduration=2.453006138 podStartE2EDuration="2.453006138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:08.45178873 +0000 UTC m=+557.581525313" watchObservedRunningTime="2026-09-15 20:46:08.453006138 +0000 UTC m=+557.582742722" Sep 15 20:46:10.064724 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:10.064686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:10.164776 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:10.164730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:16.376163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:16.376121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:16.480533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:16.480496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:18.444059 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:18.444017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:19.463609 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.463578 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp"] Sep 15 20:46:19.465974 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.465957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.468154 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.468136 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 20:46:19.468840 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.468821 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-745nq\"" Sep 15 20:46:19.468896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.468822 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 20:46:19.469185 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.469163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.469280 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.469215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.469280 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.469233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29pf\" (UniqueName: \"kubernetes.io/projected/0df72186-1448-46d2-92ec-63d3716efa72-kube-api-access-l29pf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.475186 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.475162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp"] Sep 15 20:46:19.569673 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.569633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.569673 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.569674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29pf\" (UniqueName: \"kubernetes.io/projected/0df72186-1448-46d2-92ec-63d3716efa72-kube-api-access-l29pf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.569910 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.569720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.570043 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.570025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.570120 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.570091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.577483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.577458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29pf\" (UniqueName: \"kubernetes.io/projected/0df72186-1448-46d2-92ec-63d3716efa72-kube-api-access-l29pf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:19.704056 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.704026 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:19.776601 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:19.776573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:20.047685 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:20.047662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp"] Sep 15 20:46:20.050204 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:46:20.050177 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df72186_1448_46d2_92ec_63d3716efa72.slice/crio-353526d03f6dd6b59064c0531264d572e63e747b62606ffdb690d94c7a24d988 WatchSource:0}: Error finding container 353526d03f6dd6b59064c0531264d572e63e747b62606ffdb690d94c7a24d988: Status 404 returned error can't find the container with id 353526d03f6dd6b59064c0531264d572e63e747b62606ffdb690d94c7a24d988 Sep 15 20:46:20.490860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:20.490773 2580 generic.go:356] "Generic (PLEG): container finished" podID="0df72186-1448-46d2-92ec-63d3716efa72" containerID="919379c6e41fc1e5c4d27d0db5b006f532c081bc55ff76ce8e494819b636b2de" exitCode=0 Sep 15 20:46:20.491202 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:20.490866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" event={"ID":"0df72186-1448-46d2-92ec-63d3716efa72","Type":"ContainerDied","Data":"919379c6e41fc1e5c4d27d0db5b006f532c081bc55ff76ce8e494819b636b2de"} Sep 15 20:46:20.491202 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:20.490914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" event={"ID":"0df72186-1448-46d2-92ec-63d3716efa72","Type":"ContainerStarted","Data":"353526d03f6dd6b59064c0531264d572e63e747b62606ffdb690d94c7a24d988"} Sep 15 20:46:21.502055 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:21.502024 2580 generic.go:356] "Generic (PLEG): container finished" podID="0df72186-1448-46d2-92ec-63d3716efa72" containerID="232e61a82dc7a18764c534c5e57e7f6bf01a5d106152ec08391bf6cd0907f482" exitCode=0 Sep 15 20:46:21.502340 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:21.502093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" event={"ID":"0df72186-1448-46d2-92ec-63d3716efa72","Type":"ContainerDied","Data":"232e61a82dc7a18764c534c5e57e7f6bf01a5d106152ec08391bf6cd0907f482"} Sep 15 20:46:22.508554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:22.508519 2580 generic.go:356] "Generic (PLEG): container finished" podID="0df72186-1448-46d2-92ec-63d3716efa72" containerID="9b40443256b0a676cb3b6c4d95db81c84af031839d4c29a6f2671842ed9e84d3" exitCode=0 Sep 15 20:46:22.508921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:22.508570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" event={"ID":"0df72186-1448-46d2-92ec-63d3716efa72","Type":"ContainerDied","Data":"9b40443256b0a676cb3b6c4d95db81c84af031839d4c29a6f2671842ed9e84d3"} Sep 15 20:46:23.633804 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.633777 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:23.700688 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.700657 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-bundle\") pod \"0df72186-1448-46d2-92ec-63d3716efa72\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " Sep 15 20:46:23.700819 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.700707 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-util\") pod \"0df72186-1448-46d2-92ec-63d3716efa72\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " Sep 15 20:46:23.700819 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.700731 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l29pf\" (UniqueName: \"kubernetes.io/projected/0df72186-1448-46d2-92ec-63d3716efa72-kube-api-access-l29pf\") pod \"0df72186-1448-46d2-92ec-63d3716efa72\" (UID: \"0df72186-1448-46d2-92ec-63d3716efa72\") " Sep 15 20:46:23.701348 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.701314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-bundle" (OuterVolumeSpecName: "bundle") pod "0df72186-1448-46d2-92ec-63d3716efa72" (UID: "0df72186-1448-46d2-92ec-63d3716efa72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:46:23.702750 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.702724 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df72186-1448-46d2-92ec-63d3716efa72-kube-api-access-l29pf" (OuterVolumeSpecName: "kube-api-access-l29pf") pod "0df72186-1448-46d2-92ec-63d3716efa72" (UID: "0df72186-1448-46d2-92ec-63d3716efa72"). InnerVolumeSpecName "kube-api-access-l29pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:46:23.706416 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.706384 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-util" (OuterVolumeSpecName: "util") pod "0df72186-1448-46d2-92ec-63d3716efa72" (UID: "0df72186-1448-46d2-92ec-63d3716efa72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:46:23.801313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.801254 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:23.801313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.801278 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29pf\" (UniqueName: \"kubernetes.io/projected/0df72186-1448-46d2-92ec-63d3716efa72-kube-api-access-l29pf\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:23.801313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:23.801289 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0df72186-1448-46d2-92ec-63d3716efa72-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:24.519454 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:24.519422 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" Sep 15 20:46:24.519454 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:24.519442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afht7qp" event={"ID":"0df72186-1448-46d2-92ec-63d3716efa72","Type":"ContainerDied","Data":"353526d03f6dd6b59064c0531264d572e63e747b62606ffdb690d94c7a24d988"} Sep 15 20:46:24.519662 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:24.519497 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353526d03f6dd6b59064c0531264d572e63e747b62606ffdb690d94c7a24d988" Sep 15 20:46:29.678254 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678219 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6"] Sep 15 20:46:29.678727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678707 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df72186-1448-46d2-92ec-63d3716efa72" containerName="util" Sep 15 20:46:29.678727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678726 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df72186-1448-46d2-92ec-63d3716efa72" containerName="util" Sep 15 20:46:29.678843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678751 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df72186-1448-46d2-92ec-63d3716efa72" containerName="pull" Sep 15 20:46:29.678843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678759 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df72186-1448-46d2-92ec-63d3716efa72" containerName="pull" Sep 15 20:46:29.678843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678773 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df72186-1448-46d2-92ec-63d3716efa72" containerName="extract" Sep 15 20:46:29.678843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678783 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df72186-1448-46d2-92ec-63d3716efa72" containerName="extract" Sep 15 20:46:29.679044 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.678872 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df72186-1448-46d2-92ec-63d3716efa72" containerName="extract" Sep 15 20:46:29.682293 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.682272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.684247 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.684221 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 20:46:29.684355 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.684315 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 20:46:29.684833 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.684818 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-745nq\"" Sep 15 20:46:29.691071 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.691050 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6"] Sep 15 20:46:29.744780 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.744757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gv28\" (UniqueName: \"kubernetes.io/projected/09c2665f-8af9-43af-92dc-b4ba47d747e6-kube-api-access-2gv28\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.744909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.744794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.744909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.744820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.845674 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.845646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gv28\" (UniqueName: \"kubernetes.io/projected/09c2665f-8af9-43af-92dc-b4ba47d747e6-kube-api-access-2gv28\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.845783 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.845688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.845835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.845804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.846007 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.845991 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.846168 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.846148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.853912 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.853882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gv28\" (UniqueName: \"kubernetes.io/projected/09c2665f-8af9-43af-92dc-b4ba47d747e6-kube-api-access-2gv28\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:29.991852 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:29.991793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:30.261552 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.261526 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6"] Sep 15 20:46:30.263793 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:46:30.263767 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c2665f_8af9_43af_92dc_b4ba47d747e6.slice/crio-73c2e7e3c0ad10a61bcb8e51f78e27485f127657bf5eed1753f2acc5dee840fd WatchSource:0}: Error finding container 73c2e7e3c0ad10a61bcb8e51f78e27485f127657bf5eed1753f2acc5dee840fd: Status 404 returned error can't find the container with id 73c2e7e3c0ad10a61bcb8e51f78e27485f127657bf5eed1753f2acc5dee840fd Sep 15 20:46:30.548756 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.548674 2580 generic.go:356] "Generic (PLEG): container finished" podID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerID="798cad41c498fb7cafbca3efdfcf8b3357bd16f221a64cc2208e9f130acbf574" exitCode=0 Sep 15 20:46:30.548756 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.548721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" event={"ID":"09c2665f-8af9-43af-92dc-b4ba47d747e6","Type":"ContainerDied","Data":"798cad41c498fb7cafbca3efdfcf8b3357bd16f221a64cc2208e9f130acbf574"} Sep 15 20:46:30.548927 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.548758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" event={"ID":"09c2665f-8af9-43af-92dc-b4ba47d747e6","Type":"ContainerStarted","Data":"73c2e7e3c0ad10a61bcb8e51f78e27485f127657bf5eed1753f2acc5dee840fd"} Sep 15 20:46:30.718853 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.718818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:30.779414 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.779371 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q"] Sep 15 20:46:30.782758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.782743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.785356 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.785328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 15 20:46:30.785356 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.785350 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 15 20:46:30.785583 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.785497 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 15 20:46:30.785643 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.785600 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-w6j86\"" Sep 15 20:46:30.785736 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.785722 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 15 20:46:30.785782 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.785772 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 15 20:46:30.792276 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.792246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q"] Sep 15 20:46:30.802562 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.802509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:30.854260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.854229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85d18072-74e1-4ccd-a4ba-57078cace683-manager-config\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.854392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.854278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d18072-74e1-4ccd-a4ba-57078cace683-cert\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.854392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.854312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d8c\" (UniqueName: \"kubernetes.io/projected/85d18072-74e1-4ccd-a4ba-57078cace683-kube-api-access-k6d8c\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.854392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.854330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85d18072-74e1-4ccd-a4ba-57078cace683-metrics-cert\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.954971 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.954943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d18072-74e1-4ccd-a4ba-57078cace683-cert\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.955122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.954985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d8c\" (UniqueName: \"kubernetes.io/projected/85d18072-74e1-4ccd-a4ba-57078cace683-kube-api-access-k6d8c\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.955122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.955006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85d18072-74e1-4ccd-a4ba-57078cace683-metrics-cert\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.955122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.955077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85d18072-74e1-4ccd-a4ba-57078cace683-manager-config\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.955765 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.955737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85d18072-74e1-4ccd-a4ba-57078cace683-manager-config\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.957516 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.957497 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d18072-74e1-4ccd-a4ba-57078cace683-cert\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.957597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.957578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85d18072-74e1-4ccd-a4ba-57078cace683-metrics-cert\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.963396 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.963371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d8c\" (UniqueName: \"kubernetes.io/projected/85d18072-74e1-4ccd-a4ba-57078cace683-kube-api-access-k6d8c\") pod \"lws-controller-manager-675cf9bd68-fkl5q\" (UID: \"85d18072-74e1-4ccd-a4ba-57078cace683\") " pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:30.984788 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:30.984734 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:31.084919 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.084865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:31.092663 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.092641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:31.415562 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.415497 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q"] Sep 15 20:46:31.418221 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:46:31.418195 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d18072_74e1_4ccd_a4ba_57078cace683.slice/crio-e07642e7f95694c571eafb57e65192e757e9b7c51b338a505a881e46fbba3075 WatchSource:0}: Error finding container e07642e7f95694c571eafb57e65192e757e9b7c51b338a505a881e46fbba3075: Status 404 returned error can't find the container with id e07642e7f95694c571eafb57e65192e757e9b7c51b338a505a881e46fbba3075 Sep 15 20:46:31.558126 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.558087 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" event={"ID":"85d18072-74e1-4ccd-a4ba-57078cace683","Type":"ContainerStarted","Data":"e07642e7f95694c571eafb57e65192e757e9b7c51b338a505a881e46fbba3075"} Sep 15 20:46:31.559871 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.559844 2580 generic.go:356] "Generic (PLEG): container finished" podID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerID="93f4f93f7d11e8af7b4d5cd8937a1edbd7f4caee7858339992c7ab95b5c60a0f" exitCode=0 Sep 15 20:46:31.559969 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.559931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" event={"ID":"09c2665f-8af9-43af-92dc-b4ba47d747e6","Type":"ContainerDied","Data":"93f4f93f7d11e8af7b4d5cd8937a1edbd7f4caee7858339992c7ab95b5c60a0f"} Sep 15 20:46:31.580997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.580976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:31.879198 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.879159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:31.984309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:31.984196 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:32.021494 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.021459 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr"] Sep 15 20:46:32.024982 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.024962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.027636 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.027619 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 20:46:32.027959 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.027938 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 20:46:32.028072 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.027985 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 20:46:32.028161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.028149 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 20:46:32.028613 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.028588 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fg66b\"" Sep 15 20:46:32.039372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.039347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr"] Sep 15 20:46:32.066222 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.066193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69d81a70-c5a5-4b80-9b20-cee452b4e24c-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.066318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.066226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69d81a70-c5a5-4b80-9b20-cee452b4e24c-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.066372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.066314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d81a70-c5a5-4b80-9b20-cee452b4e24c-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.066424 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.066374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92rv\" (UniqueName: \"kubernetes.io/projected/69d81a70-c5a5-4b80-9b20-cee452b4e24c-kube-api-access-x92rv\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.066424 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.066417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69d81a70-c5a5-4b80-9b20-cee452b4e24c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.130741 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.130697 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:32.167049 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.167013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69d81a70-c5a5-4b80-9b20-cee452b4e24c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.167236 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.167071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69d81a70-c5a5-4b80-9b20-cee452b4e24c-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.167236 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.167104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69d81a70-c5a5-4b80-9b20-cee452b4e24c-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.167236 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.167186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d81a70-c5a5-4b80-9b20-cee452b4e24c-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.167236 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.167224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92rv\" (UniqueName: \"kubernetes.io/projected/69d81a70-c5a5-4b80-9b20-cee452b4e24c-kube-api-access-x92rv\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.167820 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.167751 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69d81a70-c5a5-4b80-9b20-cee452b4e24c-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.170044 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.169994 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d81a70-c5a5-4b80-9b20-cee452b4e24c-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.170181 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.170158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69d81a70-c5a5-4b80-9b20-cee452b4e24c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.170480 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.170457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69d81a70-c5a5-4b80-9b20-cee452b4e24c-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.187876 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.187837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92rv\" (UniqueName: \"kubernetes.io/projected/69d81a70-c5a5-4b80-9b20-cee452b4e24c-kube-api-access-x92rv\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-4fgfr\" (UID: \"69d81a70-c5a5-4b80-9b20-cee452b4e24c\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.229928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.229896 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:32.329730 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.329693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:32.337561 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.337517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:32.431388 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.431353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:32.532354 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.532324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:32.568727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.568687 2580 generic.go:356] "Generic (PLEG): container finished" podID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerID="60b6e2234c003d991782f4314943748c2a30460e1ec228cc83a7309ab3ed3041" exitCode=0 Sep 15 20:46:32.568883 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.568777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" event={"ID":"09c2665f-8af9-43af-92dc-b4ba47d747e6","Type":"ContainerDied","Data":"60b6e2234c003d991782f4314943748c2a30460e1ec228cc83a7309ab3ed3041"} Sep 15 20:46:32.622372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:32.622348 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr"] Sep 15 20:46:32.868729 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:46:32.868698 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d81a70_c5a5_4b80_9b20_cee452b4e24c.slice/crio-41a95e367076a62888b703d8b7bb17b286b2d65cde2d173b982a09fd33d52705 WatchSource:0}: Error finding container 41a95e367076a62888b703d8b7bb17b286b2d65cde2d173b982a09fd33d52705: Status 404 returned error can't find the container with id 41a95e367076a62888b703d8b7bb17b286b2d65cde2d173b982a09fd33d52705 Sep 15 20:46:33.581941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.581895 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" event={"ID":"85d18072-74e1-4ccd-a4ba-57078cace683","Type":"ContainerStarted","Data":"f8ae548ef781e193cb5c01a6b2f9e6f02f0ed29960ebcbfc6103922805ab2ad3"} Sep 15 20:46:33.582381 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.581961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:33.583739 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.583704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" event={"ID":"69d81a70-c5a5-4b80-9b20-cee452b4e24c","Type":"ContainerStarted","Data":"41a95e367076a62888b703d8b7bb17b286b2d65cde2d173b982a09fd33d52705"} Sep 15 20:46:33.597611 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.597307 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" podStartSLOduration=2.092694531 podStartE2EDuration="3.597289214s" totalImagesPullingTime="1.504594683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:33.596355725 +0000 UTC m=+582.726092309" watchObservedRunningTime="2026-09-15 20:46:33.597289214 +0000 UTC m=+582.727025797" Sep 15 20:46:33.725643 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.725567 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:33.783331 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.783302 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-bundle\") pod \"09c2665f-8af9-43af-92dc-b4ba47d747e6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " Sep 15 20:46:33.783517 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.783376 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-util\") pod \"09c2665f-8af9-43af-92dc-b4ba47d747e6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " Sep 15 20:46:33.783517 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.783440 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gv28\" (UniqueName: \"kubernetes.io/projected/09c2665f-8af9-43af-92dc-b4ba47d747e6-kube-api-access-2gv28\") pod \"09c2665f-8af9-43af-92dc-b4ba47d747e6\" (UID: \"09c2665f-8af9-43af-92dc-b4ba47d747e6\") " Sep 15 20:46:33.784371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.784325 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-bundle" (OuterVolumeSpecName: "bundle") pod "09c2665f-8af9-43af-92dc-b4ba47d747e6" (UID: "09c2665f-8af9-43af-92dc-b4ba47d747e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:46:33.786094 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.786065 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c2665f-8af9-43af-92dc-b4ba47d747e6-kube-api-access-2gv28" (OuterVolumeSpecName: "kube-api-access-2gv28") pod "09c2665f-8af9-43af-92dc-b4ba47d747e6" (UID: "09c2665f-8af9-43af-92dc-b4ba47d747e6"). InnerVolumeSpecName "kube-api-access-2gv28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:46:33.791134 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.791108 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-util" (OuterVolumeSpecName: "util") pod "09c2665f-8af9-43af-92dc-b4ba47d747e6" (UID: "09c2665f-8af9-43af-92dc-b4ba47d747e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:46:33.884263 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.884229 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:33.884392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.884267 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gv28\" (UniqueName: \"kubernetes.io/projected/09c2665f-8af9-43af-92dc-b4ba47d747e6-kube-api-access-2gv28\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:33.884392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:33.884281 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c2665f-8af9-43af-92dc-b4ba47d747e6-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:46:34.592136 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:34.592107 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" Sep 15 20:46:34.592677 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:34.592141 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0m69s6" event={"ID":"09c2665f-8af9-43af-92dc-b4ba47d747e6","Type":"ContainerDied","Data":"73c2e7e3c0ad10a61bcb8e51f78e27485f127657bf5eed1753f2acc5dee840fd"} Sep 15 20:46:34.592677 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:34.592188 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c2e7e3c0ad10a61bcb8e51f78e27485f127657bf5eed1753f2acc5dee840fd" Sep 15 20:46:36.603039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:36.603000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" event={"ID":"69d81a70-c5a5-4b80-9b20-cee452b4e24c","Type":"ContainerStarted","Data":"c04190cbfccc853e5e0762b603bf6577b1b96aef3219ec7c934e65e338ada898"} Sep 15 20:46:37.609090 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:37.609053 2580 generic.go:356] "Generic (PLEG): container finished" podID="69d81a70-c5a5-4b80-9b20-cee452b4e24c" containerID="c04190cbfccc853e5e0762b603bf6577b1b96aef3219ec7c934e65e338ada898" exitCode=0 Sep 15 20:46:37.609488 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:37.609137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" event={"ID":"69d81a70-c5a5-4b80-9b20-cee452b4e24c","Type":"ContainerDied","Data":"c04190cbfccc853e5e0762b603bf6577b1b96aef3219ec7c934e65e338ada898"} Sep 15 20:46:38.617700 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:38.617664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" event={"ID":"69d81a70-c5a5-4b80-9b20-cee452b4e24c","Type":"ContainerStarted","Data":"fa6b6ae61bbdcf5a563bef2adc790766415c6647009373b1b3638c08c4dd4c4b"} Sep 15 20:46:38.618092 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:38.617756 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:38.636976 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:38.636922 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" podStartSLOduration=3.359185296 podStartE2EDuration="7.636901018s" totalImagesPullingTime="3.181751253s" totalInitContainerRuntime="1.095964469s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:38.635686894 +0000 UTC m=+587.765423483" watchObservedRunningTime="2026-09-15 20:46:38.636901018 +0000 UTC m=+587.766637603" Sep 15 20:46:42.455017 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:42.454975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:42.554554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:42.554521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:44.594266 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:44.594235 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-675cf9bd68-fkl5q" Sep 15 20:46:47.612029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:47.611997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:49.209720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:49.209683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:49.624531 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:49.624501 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-4fgfr" Sep 15 20:46:51.251845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:51.251813 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:46:51.252773 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:51.252754 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:46:52.057336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:52.057295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:52.153304 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:52.153272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:52.411334 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:52.411247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:52.579569 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:52.579534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:53.577179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:53.577147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:53.678068 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:53.678034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:53.746827 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:53.746794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:53.806299 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:53.806264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:54.779700 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:54.779663 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:55.610913 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:55.610880 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:58.162291 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.162249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:58.265299 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.265269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:58.713122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713091 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l"] Sep 15 20:46:58.713463 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713449 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerName="pull" Sep 15 20:46:58.713522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713465 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerName="pull" Sep 15 20:46:58.713522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713474 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerName="util" Sep 15 20:46:58.713522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713479 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerName="util" Sep 15 20:46:58.713522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713500 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerName="extract" Sep 15 20:46:58.713522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713506 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerName="extract" Sep 15 20:46:58.713717 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.713560 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c2665f-8af9-43af-92dc-b4ba47d747e6" containerName="extract" Sep 15 20:46:58.716461 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.716433 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.718697 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.718664 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rx7vx\"" Sep 15 20:46:58.718817 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.718741 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 20:46:58.729232 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.729207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l"] Sep 15 20:46:58.747634 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.747605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:58.797710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.797823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.797823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.797823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.797983 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.797983 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcrc\" (UniqueName: \"kubernetes.io/projected/1364f2e9-1cb6-4810-9daf-99fd905f06ab-kube-api-access-qvcrc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.797983 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.797983 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.797924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.798117 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.798025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.898866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.898830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.898886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.898915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcrc\" (UniqueName: \"kubernetes.io/projected/1364f2e9-1cb6-4810-9daf-99fd905f06ab-kube-api-access-qvcrc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.898934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.898955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.898984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899375 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899375 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899524 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899524 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899601 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899601 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.899816 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.899794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.901867 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.901838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.902072 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.902010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.911491 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.911468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1364f2e9-1cb6-4810-9daf-99fd905f06ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.911857 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.911833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcrc\" (UniqueName: \"kubernetes.io/projected/1364f2e9-1cb6-4810-9daf-99fd905f06ab-kube-api-access-qvcrc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l\" (UID: \"1364f2e9-1cb6-4810-9daf-99fd905f06ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:58.932606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:58.932579 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:59.027217 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:59.027189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:46:59.032823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:59.032795 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:46:59.304181 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:59.304153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l"] Sep 15 20:46:59.306319 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:46:59.306291 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1364f2e9_1cb6_4810_9daf_99fd905f06ab.slice/crio-b557a7fc3dac8b55644d531b4c335ce9abdd425d7c494bf8054a290f124010b2 WatchSource:0}: Error finding container b557a7fc3dac8b55644d531b4c335ce9abdd425d7c494bf8054a290f124010b2: Status 404 returned error can't find the container with id b557a7fc3dac8b55644d531b4c335ce9abdd425d7c494bf8054a290f124010b2 Sep 15 20:46:59.724081 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:46:59.724001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" event={"ID":"1364f2e9-1cb6-4810-9daf-99fd905f06ab","Type":"ContainerStarted","Data":"b557a7fc3dac8b55644d531b4c335ce9abdd425d7c494bf8054a290f124010b2"} Sep 15 20:47:01.972505 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:01.972466 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:47:01.972736 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:01.972544 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:47:01.972736 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:01.972577 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:47:02.742194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:02.742154 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" event={"ID":"1364f2e9-1cb6-4810-9daf-99fd905f06ab","Type":"ContainerStarted","Data":"9996b23a889a5111c830dee532d99dd31241800796534f50b946c8e6750a446c"} Sep 15 20:47:02.760081 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:02.760026 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" podStartSLOduration=2.09637531 podStartE2EDuration="4.760009535s" totalImagesPullingTime="2.663634225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:47:02.758816718 +0000 UTC m=+611.888553302" watchObservedRunningTime="2026-09-15 20:47:02.760009535 +0000 UTC m=+611.889746118" Sep 15 20:47:03.027678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.027641 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:47:03.032691 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.032661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:47:03.132729 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.132691 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-w7hvt"] Sep 15 20:47:03.137934 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.137907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.140177 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.140149 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ppt24\"" Sep 15 20:47:03.140601 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.140581 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 15 20:47:03.152010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.151982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-w7hvt"] Sep 15 20:47:03.238305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.238261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3a611af-d53e-409c-a463-63b98690a3b2-cert\") pod \"odh-model-controller-7998c67dc4-w7hvt\" (UID: \"d3a611af-d53e-409c-a463-63b98690a3b2\") " pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.238305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.238298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdh7\" (UniqueName: \"kubernetes.io/projected/d3a611af-d53e-409c-a463-63b98690a3b2-kube-api-access-wkdh7\") pod \"odh-model-controller-7998c67dc4-w7hvt\" (UID: \"d3a611af-d53e-409c-a463-63b98690a3b2\") " pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.326623 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.326534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:03.339491 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.339458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3a611af-d53e-409c-a463-63b98690a3b2-cert\") pod \"odh-model-controller-7998c67dc4-w7hvt\" (UID: \"d3a611af-d53e-409c-a463-63b98690a3b2\") " pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.339685 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.339505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdh7\" (UniqueName: \"kubernetes.io/projected/d3a611af-d53e-409c-a463-63b98690a3b2-kube-api-access-wkdh7\") pod \"odh-model-controller-7998c67dc4-w7hvt\" (UID: \"d3a611af-d53e-409c-a463-63b98690a3b2\") " pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.339758 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:47:03.339683 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 15 20:47:03.339817 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:47:03.339757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a611af-d53e-409c-a463-63b98690a3b2-cert podName:d3a611af-d53e-409c-a463-63b98690a3b2 nodeName:}" failed. No retries permitted until 2026-09-15 20:47:03.839736334 +0000 UTC m=+612.969472907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d3a611af-d53e-409c-a463-63b98690a3b2-cert") pod "odh-model-controller-7998c67dc4-w7hvt" (UID: "d3a611af-d53e-409c-a463-63b98690a3b2") : secret "odh-model-controller-webhook-cert" not found Sep 15 20:47:03.347987 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.347960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdh7\" (UniqueName: \"kubernetes.io/projected/d3a611af-d53e-409c-a463-63b98690a3b2-kube-api-access-wkdh7\") pod \"odh-model-controller-7998c67dc4-w7hvt\" (UID: \"d3a611af-d53e-409c-a463-63b98690a3b2\") " pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.436180 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.436142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:03.666629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.666519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:03.685720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.685690 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2p2gd"] Sep 15 20:47:03.688968 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.688949 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:03.691366 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.691341 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 15 20:47:03.701250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.701224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2p2gd"] Sep 15 20:47:03.735941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.735901 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:03.748332 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.748305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:47:03.749283 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.749260 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l" Sep 15 20:47:03.843740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.843703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef9dcd-5b1c-487b-a779-b551c321365c-tls-certs\") pod \"model-serving-api-54d89bd5fc-2p2gd\" (UID: \"b8ef9dcd-5b1c-487b-a779-b551c321365c\") " pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:03.843926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.843754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbh9\" (UniqueName: \"kubernetes.io/projected/b8ef9dcd-5b1c-487b-a779-b551c321365c-kube-api-access-rjbh9\") pod \"model-serving-api-54d89bd5fc-2p2gd\" (UID: \"b8ef9dcd-5b1c-487b-a779-b551c321365c\") " pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:03.843926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.843914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3a611af-d53e-409c-a463-63b98690a3b2-cert\") pod \"odh-model-controller-7998c67dc4-w7hvt\" (UID: \"d3a611af-d53e-409c-a463-63b98690a3b2\") " pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.846183 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.846159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3a611af-d53e-409c-a463-63b98690a3b2-cert\") pod \"odh-model-controller-7998c67dc4-w7hvt\" (UID: \"d3a611af-d53e-409c-a463-63b98690a3b2\") " pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:03.929997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.929913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:03.945195 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.945160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef9dcd-5b1c-487b-a779-b551c321365c-tls-certs\") pod \"model-serving-api-54d89bd5fc-2p2gd\" (UID: \"b8ef9dcd-5b1c-487b-a779-b551c321365c\") " pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:03.945195 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.945194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbh9\" (UniqueName: \"kubernetes.io/projected/b8ef9dcd-5b1c-487b-a779-b551c321365c-kube-api-access-rjbh9\") pod \"model-serving-api-54d89bd5fc-2p2gd\" (UID: \"b8ef9dcd-5b1c-487b-a779-b551c321365c\") " pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:03.947694 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.947662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef9dcd-5b1c-487b-a779-b551c321365c-tls-certs\") pod \"model-serving-api-54d89bd5fc-2p2gd\" (UID: \"b8ef9dcd-5b1c-487b-a779-b551c321365c\") " pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:03.952862 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.952838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbh9\" (UniqueName: \"kubernetes.io/projected/b8ef9dcd-5b1c-487b-a779-b551c321365c-kube-api-access-rjbh9\") pod \"model-serving-api-54d89bd5fc-2p2gd\" (UID: \"b8ef9dcd-5b1c-487b-a779-b551c321365c\") " pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:03.999144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:03.999117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:04.053714 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:04.053248 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:04.271970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:04.271945 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2p2gd"] Sep 15 20:47:04.276719 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:47:04.276681 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ef9dcd_5b1c_487b_a779_b551c321365c.slice/crio-a334789860de92ff2afeefb0c3698ac91f2ad3b15ca7cfe7d011b0323281cefa WatchSource:0}: Error finding container a334789860de92ff2afeefb0c3698ac91f2ad3b15ca7cfe7d011b0323281cefa: Status 404 returned error can't find the container with id a334789860de92ff2afeefb0c3698ac91f2ad3b15ca7cfe7d011b0323281cefa Sep 15 20:47:04.331900 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:04.331872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-w7hvt"] Sep 15 20:47:04.333818 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:47:04.333792 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a611af_d53e_409c_a463_63b98690a3b2.slice/crio-15b97820277609d6af22a1b66af0dfc2d59ef256b35467d8159005f5028cf41f WatchSource:0}: Error finding container 15b97820277609d6af22a1b66af0dfc2d59ef256b35467d8159005f5028cf41f: Status 404 returned error can't find the container with id 15b97820277609d6af22a1b66af0dfc2d59ef256b35467d8159005f5028cf41f Sep 15 20:47:04.754986 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:04.754942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" event={"ID":"d3a611af-d53e-409c-a463-63b98690a3b2","Type":"ContainerStarted","Data":"15b97820277609d6af22a1b66af0dfc2d59ef256b35467d8159005f5028cf41f"} Sep 15 20:47:04.756460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:04.756426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" event={"ID":"b8ef9dcd-5b1c-487b-a779-b551c321365c","Type":"ContainerStarted","Data":"a334789860de92ff2afeefb0c3698ac91f2ad3b15ca7cfe7d011b0323281cefa"} Sep 15 20:47:06.581732 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:06.581647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:06.816323 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:06.816289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:06.866347 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:06.866212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:09.790675 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:09.790627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" event={"ID":"d3a611af-d53e-409c-a463-63b98690a3b2","Type":"ContainerStarted","Data":"e25b18ae5967e29b5aeb5a938dc64083c08a2d199a16d7e11132804c74fc720c"} Sep 15 20:47:09.791150 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:09.790717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:09.792181 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:09.792151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" event={"ID":"b8ef9dcd-5b1c-487b-a779-b551c321365c","Type":"ContainerStarted","Data":"2c4f98e711e4da946c9620ba4a40b2b4ab4754286f8d965ea729bf3e0b4ad8a5"} Sep 15 20:47:09.792303 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:09.792289 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:09.809388 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:09.809347 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" podStartSLOduration=2.388691191 podStartE2EDuration="6.809334906s" totalImagesPullingTime="4.420643715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:47:09.807376715 +0000 UTC m=+618.937113291" watchObservedRunningTime="2026-09-15 20:47:09.809334906 +0000 UTC m=+618.939071515" Sep 15 20:47:09.822979 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:09.822936 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" podStartSLOduration=2.3024785850000002 podStartE2EDuration="6.822920617s" totalImagesPullingTime="4.520442032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:47:09.822508746 +0000 UTC m=+618.952245329" watchObservedRunningTime="2026-09-15 20:47:09.822920617 +0000 UTC m=+618.952657200" Sep 15 20:47:13.409240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.409203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:13.491049 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.491014 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g"] Sep 15 20:47:13.495237 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.495196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:13.497596 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.497567 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hk54x\"" Sep 15 20:47:13.497728 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.497692 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 15 20:47:13.505660 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.505636 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g"] Sep 15 20:47:13.511860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.511824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:13.541353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.541326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzhp\" (UniqueName: \"kubernetes.io/projected/f04b6918-b1ee-41ed-989f-a187aee1e26a-kube-api-access-6hzhp\") pod \"llmisvc-controller-manager-777c6bd7d5-8bg5g\" (UID: \"f04b6918-b1ee-41ed-989f-a187aee1e26a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:13.541494 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.541360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f04b6918-b1ee-41ed-989f-a187aee1e26a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bg5g\" (UID: \"f04b6918-b1ee-41ed-989f-a187aee1e26a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:13.596881 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.596824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:13.642294 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.642265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzhp\" (UniqueName: \"kubernetes.io/projected/f04b6918-b1ee-41ed-989f-a187aee1e26a-kube-api-access-6hzhp\") pod \"llmisvc-controller-manager-777c6bd7d5-8bg5g\" (UID: \"f04b6918-b1ee-41ed-989f-a187aee1e26a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:13.642457 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.642300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f04b6918-b1ee-41ed-989f-a187aee1e26a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bg5g\" (UID: \"f04b6918-b1ee-41ed-989f-a187aee1e26a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:13.642514 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:47:13.642463 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 15 20:47:13.642560 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:47:13.642515 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f04b6918-b1ee-41ed-989f-a187aee1e26a-cert podName:f04b6918-b1ee-41ed-989f-a187aee1e26a nodeName:}" failed. No retries permitted until 2026-09-15 20:47:14.142499624 +0000 UTC m=+623.272236184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f04b6918-b1ee-41ed-989f-a187aee1e26a-cert") pod "llmisvc-controller-manager-777c6bd7d5-8bg5g" (UID: "f04b6918-b1ee-41ed-989f-a187aee1e26a") : secret "llmisvc-webhook-server-cert" not found Sep 15 20:47:13.654613 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.654578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzhp\" (UniqueName: \"kubernetes.io/projected/f04b6918-b1ee-41ed-989f-a187aee1e26a-kube-api-access-6hzhp\") pod \"llmisvc-controller-manager-777c6bd7d5-8bg5g\" (UID: \"f04b6918-b1ee-41ed-989f-a187aee1e26a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:13.815128 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.815096 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:13.915613 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:13.915578 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:14.154169 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.146048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f04b6918-b1ee-41ed-989f-a187aee1e26a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bg5g\" (UID: \"f04b6918-b1ee-41ed-989f-a187aee1e26a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:14.154169 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.149276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f04b6918-b1ee-41ed-989f-a187aee1e26a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bg5g\" (UID: \"f04b6918-b1ee-41ed-989f-a187aee1e26a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:14.405956 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.405873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:14.670184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.670105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:14.676938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.676915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g"] Sep 15 20:47:14.679189 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:47:14.679163 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf04b6918_b1ee_41ed_989f_a187aee1e26a.slice/crio-34d65ceeb6944d2c1e4d85fff92ad61c57a7d2bed550c6b8e01cf9158b7328ef WatchSource:0}: Error finding container 34d65ceeb6944d2c1e4d85fff92ad61c57a7d2bed550c6b8e01cf9158b7328ef: Status 404 returned error can't find the container with id 34d65ceeb6944d2c1e4d85fff92ad61c57a7d2bed550c6b8e01cf9158b7328ef Sep 15 20:47:14.680563 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.680544 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:47:14.787052 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.787021 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:14.816999 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.816973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" event={"ID":"f04b6918-b1ee-41ed-989f-a187aee1e26a","Type":"ContainerStarted","Data":"34d65ceeb6944d2c1e4d85fff92ad61c57a7d2bed550c6b8e01cf9158b7328ef"} Sep 15 20:47:14.895699 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:14.895672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:18.848365 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:18.848322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" event={"ID":"f04b6918-b1ee-41ed-989f-a187aee1e26a","Type":"ContainerStarted","Data":"23408fbdd9bd53cdf5346460764a42e604dd62c74442ba1e208804c56929b26e"} Sep 15 20:47:18.848906 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:18.848378 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:47:18.865228 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:18.865181 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" podStartSLOduration=2.769072368 podStartE2EDuration="5.86516578s" totalImagesPullingTime="3.096093412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:47:18.863710497 +0000 UTC m=+627.993447080" watchObservedRunningTime="2026-09-15 20:47:18.86516578 +0000 UTC m=+627.994902372" Sep 15 20:47:19.716040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:19.716004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:19.770165 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:19.770133 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:20.799138 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:20.799104 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-w7hvt" Sep 15 20:47:20.801296 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:20.801275 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2p2gd" Sep 15 20:47:24.052460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:24.052423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:24.150428 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:24.150371 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:24.198531 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:24.198499 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:47:49.855052 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:47:49.855021 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bg5g" Sep 15 20:48:04.434757 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:04.434719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:04.534301 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:04.534264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:08.079610 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.079569 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98"] Sep 15 20:48:08.087511 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.087484 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.089967 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.089940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-745nq\"" Sep 15 20:48:08.090199 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.090178 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 20:48:08.090354 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.090336 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 20:48:08.090578 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.090554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98"] Sep 15 20:48:08.173828 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.173795 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd"] Sep 15 20:48:08.177701 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.177681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.185593 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.185572 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd"] Sep 15 20:48:08.198931 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.198907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.199066 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.198979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6rt\" (UniqueName: \"kubernetes.io/projected/121f4680-1852-4e8c-a460-ad9fff98915c-kube-api-access-dq6rt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.199066 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.199014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.279660 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.279631 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj"] Sep 15 20:48:08.283470 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.283452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.291298 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.291276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj"] Sep 15 20:48:08.299528 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.299501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.299657 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.299553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnkx\" (UniqueName: \"kubernetes.io/projected/c840834d-eb50-4d56-868e-2da7582e9f0a-kube-api-access-bvnkx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.299657 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.299589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.299657 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.299635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.299852 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.299823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6rt\" (UniqueName: \"kubernetes.io/projected/121f4680-1852-4e8c-a460-ad9fff98915c-kube-api-access-dq6rt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.299922 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.299906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.299999 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.299983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.300260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.300240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.307755 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.307734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6rt\" (UniqueName: \"kubernetes.io/projected/121f4680-1852-4e8c-a460-ad9fff98915c-kube-api-access-dq6rt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.377125 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.377066 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx"] Sep 15 20:48:08.380710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.380695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.389102 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.389081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx"] Sep 15 20:48:08.400731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.400707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnkx\" (UniqueName: \"kubernetes.io/projected/c840834d-eb50-4d56-868e-2da7582e9f0a-kube-api-access-bvnkx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.400843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.400739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.400843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.400768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.400843 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.400789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.401019 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.400895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8gcc\" (UniqueName: \"kubernetes.io/projected/cddba387-5f4f-46db-99b1-17dfb64ab6d3-kube-api-access-t8gcc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.401019 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.400961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.401093 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.401067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.401125 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.401096 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.404637 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.404621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:08.408845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.408826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnkx\" (UniqueName: \"kubernetes.io/projected/c840834d-eb50-4d56-868e-2da7582e9f0a-kube-api-access-bvnkx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.487523 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.487492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:08.502335 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.502425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2tb\" (UniqueName: \"kubernetes.io/projected/6db47983-124e-476c-bfe9-c29c849c9732-kube-api-access-vg2tb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.502425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.502544 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.502597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8gcc\" (UniqueName: \"kubernetes.io/projected/cddba387-5f4f-46db-99b1-17dfb64ab6d3-kube-api-access-t8gcc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.502657 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.502790 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.502949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.502929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.510200 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.510175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8gcc\" (UniqueName: \"kubernetes.io/projected/cddba387-5f4f-46db-99b1-17dfb64ab6d3-kube-api-access-t8gcc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.592933 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.592897 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:08.603570 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.603543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.603665 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.603636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.603734 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.603664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2tb\" (UniqueName: \"kubernetes.io/projected/6db47983-124e-476c-bfe9-c29c849c9732-kube-api-access-vg2tb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.603970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.603946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.604071 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.604051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.612529 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.612492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2tb\" (UniqueName: \"kubernetes.io/projected/6db47983-124e-476c-bfe9-c29c849c9732-kube-api-access-vg2tb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.677667 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.677597 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98"] Sep 15 20:48:08.680651 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:48:08.680622 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121f4680_1852_4e8c_a460_ad9fff98915c.slice/crio-ba58e3aecdae63f64b42ae4e4d6015e85114e670d83ca56b5b7b8013a951ed20 WatchSource:0}: Error finding container ba58e3aecdae63f64b42ae4e4d6015e85114e670d83ca56b5b7b8013a951ed20: Status 404 returned error can't find the container with id ba58e3aecdae63f64b42ae4e4d6015e85114e670d83ca56b5b7b8013a951ed20 Sep 15 20:48:08.690871 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.690847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:08.761815 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.761785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd"] Sep 15 20:48:08.763642 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:48:08.763616 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc840834d_eb50_4d56_868e_2da7582e9f0a.slice/crio-a1e64d32c1147033746b8ff80279edebc990bd5384d0d279955c6cf2b89c3925 WatchSource:0}: Error finding container a1e64d32c1147033746b8ff80279edebc990bd5384d0d279955c6cf2b89c3925: Status 404 returned error can't find the container with id a1e64d32c1147033746b8ff80279edebc990bd5384d0d279955c6cf2b89c3925 Sep 15 20:48:08.871415 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.871375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj"] Sep 15 20:48:08.873087 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:48:08.873057 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddba387_5f4f_46db_99b1_17dfb64ab6d3.slice/crio-79ce8f5b3d7ec6cba7a24fc8bc76857d5d3635948743cc1afb3a466b71052814 WatchSource:0}: Error finding container 79ce8f5b3d7ec6cba7a24fc8bc76857d5d3635948743cc1afb3a466b71052814: Status 404 returned error can't find the container with id 79ce8f5b3d7ec6cba7a24fc8bc76857d5d3635948743cc1afb3a466b71052814 Sep 15 20:48:08.966834 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:08.966804 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx"] Sep 15 20:48:08.980225 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:48:08.980196 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db47983_124e_476c_bfe9_c29c849c9732.slice/crio-5d890627c94ccb3cd7b4316d903e7b4c3c8c216b2df364d72bbb56b709a55adf WatchSource:0}: Error finding container 5d890627c94ccb3cd7b4316d903e7b4c3c8c216b2df364d72bbb56b709a55adf: Status 404 returned error can't find the container with id 5d890627c94ccb3cd7b4316d903e7b4c3c8c216b2df364d72bbb56b709a55adf Sep 15 20:48:09.101270 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.101240 2580 generic.go:356] "Generic (PLEG): container finished" podID="121f4680-1852-4e8c-a460-ad9fff98915c" containerID="8f750fd19b29a80571345128df2dffe5fcd16d342f53a1853bd26ad7613c9cac" exitCode=0 Sep 15 20:48:09.101629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.101307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" event={"ID":"121f4680-1852-4e8c-a460-ad9fff98915c","Type":"ContainerDied","Data":"8f750fd19b29a80571345128df2dffe5fcd16d342f53a1853bd26ad7613c9cac"} Sep 15 20:48:09.101629 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.101354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" event={"ID":"121f4680-1852-4e8c-a460-ad9fff98915c","Type":"ContainerStarted","Data":"ba58e3aecdae63f64b42ae4e4d6015e85114e670d83ca56b5b7b8013a951ed20"} Sep 15 20:48:09.103330 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.103301 2580 generic.go:356] "Generic (PLEG): container finished" podID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerID="1a39f279a8566bf9adfbe5b453d57915410eb79bc97a43d86c379577429f1cfc" exitCode=0 Sep 15 20:48:09.103482 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.103447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" event={"ID":"cddba387-5f4f-46db-99b1-17dfb64ab6d3","Type":"ContainerDied","Data":"1a39f279a8566bf9adfbe5b453d57915410eb79bc97a43d86c379577429f1cfc"} Sep 15 20:48:09.103651 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.103519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" event={"ID":"cddba387-5f4f-46db-99b1-17dfb64ab6d3","Type":"ContainerStarted","Data":"79ce8f5b3d7ec6cba7a24fc8bc76857d5d3635948743cc1afb3a466b71052814"} Sep 15 20:48:09.105306 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.105282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" event={"ID":"6db47983-124e-476c-bfe9-c29c849c9732","Type":"ContainerStarted","Data":"22a0a5b80e053d4d7a1299cb67e6ec6c26996e61899055b61626459f5ea312e6"} Sep 15 20:48:09.105398 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.105327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" event={"ID":"6db47983-124e-476c-bfe9-c29c849c9732","Type":"ContainerStarted","Data":"5d890627c94ccb3cd7b4316d903e7b4c3c8c216b2df364d72bbb56b709a55adf"} Sep 15 20:48:09.107120 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.107097 2580 generic.go:356] "Generic (PLEG): container finished" podID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerID="51eece15b543afdced05f9a0f05d1347e6d3292bacaf5518b277c376b6ab5f07" exitCode=0 Sep 15 20:48:09.107214 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.107171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" event={"ID":"c840834d-eb50-4d56-868e-2da7582e9f0a","Type":"ContainerDied","Data":"51eece15b543afdced05f9a0f05d1347e6d3292bacaf5518b277c376b6ab5f07"} Sep 15 20:48:09.107214 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:09.107211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" event={"ID":"c840834d-eb50-4d56-868e-2da7582e9f0a","Type":"ContainerStarted","Data":"a1e64d32c1147033746b8ff80279edebc990bd5384d0d279955c6cf2b89c3925"} Sep 15 20:48:10.113326 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:10.113296 2580 generic.go:356] "Generic (PLEG): container finished" podID="6db47983-124e-476c-bfe9-c29c849c9732" containerID="22a0a5b80e053d4d7a1299cb67e6ec6c26996e61899055b61626459f5ea312e6" exitCode=0 Sep 15 20:48:10.113653 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:10.113392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" event={"ID":"6db47983-124e-476c-bfe9-c29c849c9732","Type":"ContainerDied","Data":"22a0a5b80e053d4d7a1299cb67e6ec6c26996e61899055b61626459f5ea312e6"} Sep 15 20:48:11.127977 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:11.127942 2580 generic.go:356] "Generic (PLEG): container finished" podID="121f4680-1852-4e8c-a460-ad9fff98915c" containerID="c89cd47ab74386c3b6cf0fb95763ae2af08479ed145335f94b8fc0a8afaeaadc" exitCode=0 Sep 15 20:48:11.128378 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:11.128029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" event={"ID":"121f4680-1852-4e8c-a460-ad9fff98915c","Type":"ContainerDied","Data":"c89cd47ab74386c3b6cf0fb95763ae2af08479ed145335f94b8fc0a8afaeaadc"} Sep 15 20:48:11.130192 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:11.130168 2580 generic.go:356] "Generic (PLEG): container finished" podID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerID="a52520caf1437aba4f4bcd3fae6b325288817040b2b05097bf85ab2274797c1e" exitCode=0 Sep 15 20:48:11.130286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:11.130275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" event={"ID":"cddba387-5f4f-46db-99b1-17dfb64ab6d3","Type":"ContainerDied","Data":"a52520caf1437aba4f4bcd3fae6b325288817040b2b05097bf85ab2274797c1e"} Sep 15 20:48:11.132275 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:11.132253 2580 generic.go:356] "Generic (PLEG): container finished" podID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerID="04a57cb80445353e0b8dfe5f799151d929d01d5eb6901e6f9b756aa0c395ad22" exitCode=0 Sep 15 20:48:11.132359 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:11.132291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" event={"ID":"c840834d-eb50-4d56-868e-2da7582e9f0a","Type":"ContainerDied","Data":"04a57cb80445353e0b8dfe5f799151d929d01d5eb6901e6f9b756aa0c395ad22"} Sep 15 20:48:12.139789 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.139701 2580 generic.go:356] "Generic (PLEG): container finished" podID="121f4680-1852-4e8c-a460-ad9fff98915c" containerID="15e80c2893370d75796157b6b6dc55a18df308dc21f5190287dea05bddd50994" exitCode=0 Sep 15 20:48:12.140137 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.139824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" event={"ID":"121f4680-1852-4e8c-a460-ad9fff98915c","Type":"ContainerDied","Data":"15e80c2893370d75796157b6b6dc55a18df308dc21f5190287dea05bddd50994"} Sep 15 20:48:12.141796 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.141775 2580 generic.go:356] "Generic (PLEG): container finished" podID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerID="bac4e6638acd32dc53dc549bc4a9157a5a9494efb5c1a81aac4341f9640b77bb" exitCode=0 Sep 15 20:48:12.141906 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.141856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" event={"ID":"cddba387-5f4f-46db-99b1-17dfb64ab6d3","Type":"ContainerDied","Data":"bac4e6638acd32dc53dc549bc4a9157a5a9494efb5c1a81aac4341f9640b77bb"} Sep 15 20:48:12.143659 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.143633 2580 generic.go:356] "Generic (PLEG): container finished" podID="6db47983-124e-476c-bfe9-c29c849c9732" containerID="a6816e27f9ed6861200f865101feac62e83ac50b173be19046227a01c0462b6b" exitCode=0 Sep 15 20:48:12.143773 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.143706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" event={"ID":"6db47983-124e-476c-bfe9-c29c849c9732","Type":"ContainerDied","Data":"a6816e27f9ed6861200f865101feac62e83ac50b173be19046227a01c0462b6b"} Sep 15 20:48:12.145879 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.145857 2580 generic.go:356] "Generic (PLEG): container finished" podID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerID="fb9c7b60e63d99628af52dfb4c65d38c2d10f38028723788bd441593b358690f" exitCode=0 Sep 15 20:48:12.145970 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:12.145945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" event={"ID":"c840834d-eb50-4d56-868e-2da7582e9f0a","Type":"ContainerDied","Data":"fb9c7b60e63d99628af52dfb4c65d38c2d10f38028723788bd441593b358690f"} Sep 15 20:48:13.153681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.153645 2580 generic.go:356] "Generic (PLEG): container finished" podID="6db47983-124e-476c-bfe9-c29c849c9732" containerID="2d389cc60aabc51faf36f1271b6f43e2e22ecfc2e5ec1a52ed1eae17607275a4" exitCode=0 Sep 15 20:48:13.154085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.153734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" event={"ID":"6db47983-124e-476c-bfe9-c29c849c9732","Type":"ContainerDied","Data":"2d389cc60aabc51faf36f1271b6f43e2e22ecfc2e5ec1a52ed1eae17607275a4"} Sep 15 20:48:13.305322 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.305294 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:13.346747 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.346724 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:13.349865 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.349847 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:13.446692 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446609 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-util\") pod \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " Sep 15 20:48:13.446692 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446645 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-bundle\") pod \"c840834d-eb50-4d56-868e-2da7582e9f0a\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " Sep 15 20:48:13.446692 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446678 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvnkx\" (UniqueName: \"kubernetes.io/projected/c840834d-eb50-4d56-868e-2da7582e9f0a-kube-api-access-bvnkx\") pod \"c840834d-eb50-4d56-868e-2da7582e9f0a\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " Sep 15 20:48:13.446971 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446735 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-util\") pod \"121f4680-1852-4e8c-a460-ad9fff98915c\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " Sep 15 20:48:13.446971 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446752 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-bundle\") pod \"121f4680-1852-4e8c-a460-ad9fff98915c\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " Sep 15 20:48:13.446971 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446785 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8gcc\" (UniqueName: \"kubernetes.io/projected/cddba387-5f4f-46db-99b1-17dfb64ab6d3-kube-api-access-t8gcc\") pod \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " Sep 15 20:48:13.446971 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446898 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-bundle\") pod \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\" (UID: \"cddba387-5f4f-46db-99b1-17dfb64ab6d3\") " Sep 15 20:48:13.446971 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.446968 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-util\") pod \"c840834d-eb50-4d56-868e-2da7582e9f0a\" (UID: \"c840834d-eb50-4d56-868e-2da7582e9f0a\") " Sep 15 20:48:13.447235 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.447027 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq6rt\" (UniqueName: \"kubernetes.io/projected/121f4680-1852-4e8c-a460-ad9fff98915c-kube-api-access-dq6rt\") pod \"121f4680-1852-4e8c-a460-ad9fff98915c\" (UID: \"121f4680-1852-4e8c-a460-ad9fff98915c\") " Sep 15 20:48:13.447508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.447316 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-bundle" (OuterVolumeSpecName: "bundle") pod "c840834d-eb50-4d56-868e-2da7582e9f0a" (UID: "c840834d-eb50-4d56-868e-2da7582e9f0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:13.447508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.447382 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.447508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.447315 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-bundle" (OuterVolumeSpecName: "bundle") pod "121f4680-1852-4e8c-a460-ad9fff98915c" (UID: "121f4680-1852-4e8c-a460-ad9fff98915c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:13.448768 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.447963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-bundle" (OuterVolumeSpecName: "bundle") pod "cddba387-5f4f-46db-99b1-17dfb64ab6d3" (UID: "cddba387-5f4f-46db-99b1-17dfb64ab6d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:13.449063 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.449039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddba387-5f4f-46db-99b1-17dfb64ab6d3-kube-api-access-t8gcc" (OuterVolumeSpecName: "kube-api-access-t8gcc") pod "cddba387-5f4f-46db-99b1-17dfb64ab6d3" (UID: "cddba387-5f4f-46db-99b1-17dfb64ab6d3"). InnerVolumeSpecName "kube-api-access-t8gcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:48:13.449253 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.449205 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c840834d-eb50-4d56-868e-2da7582e9f0a-kube-api-access-bvnkx" (OuterVolumeSpecName: "kube-api-access-bvnkx") pod "c840834d-eb50-4d56-868e-2da7582e9f0a" (UID: "c840834d-eb50-4d56-868e-2da7582e9f0a"). InnerVolumeSpecName "kube-api-access-bvnkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:48:13.449627 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.449604 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121f4680-1852-4e8c-a460-ad9fff98915c-kube-api-access-dq6rt" (OuterVolumeSpecName: "kube-api-access-dq6rt") pod "121f4680-1852-4e8c-a460-ad9fff98915c" (UID: "121f4680-1852-4e8c-a460-ad9fff98915c"). InnerVolumeSpecName "kube-api-access-dq6rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:48:13.453954 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.453930 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-util" (OuterVolumeSpecName: "util") pod "121f4680-1852-4e8c-a460-ad9fff98915c" (UID: "121f4680-1852-4e8c-a460-ad9fff98915c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:13.455731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.455705 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-util" (OuterVolumeSpecName: "util") pod "cddba387-5f4f-46db-99b1-17dfb64ab6d3" (UID: "cddba387-5f4f-46db-99b1-17dfb64ab6d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:13.456260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.456237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-util" (OuterVolumeSpecName: "util") pod "c840834d-eb50-4d56-868e-2da7582e9f0a" (UID: "c840834d-eb50-4d56-868e-2da7582e9f0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:13.548035 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548014 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.548035 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548035 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/121f4680-1852-4e8c-a460-ad9fff98915c-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.548139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548044 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8gcc\" (UniqueName: \"kubernetes.io/projected/cddba387-5f4f-46db-99b1-17dfb64ab6d3-kube-api-access-t8gcc\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.548139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548054 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.548139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548062 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c840834d-eb50-4d56-868e-2da7582e9f0a-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.548139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548070 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6rt\" (UniqueName: \"kubernetes.io/projected/121f4680-1852-4e8c-a460-ad9fff98915c-kube-api-access-dq6rt\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.548139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548078 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cddba387-5f4f-46db-99b1-17dfb64ab6d3-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:13.548139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:13.548087 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnkx\" (UniqueName: \"kubernetes.io/projected/c840834d-eb50-4d56-868e-2da7582e9f0a-kube-api-access-bvnkx\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:14.161384 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.161351 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" Sep 15 20:48:14.161777 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.161384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m2snj" event={"ID":"cddba387-5f4f-46db-99b1-17dfb64ab6d3","Type":"ContainerDied","Data":"79ce8f5b3d7ec6cba7a24fc8bc76857d5d3635948743cc1afb3a466b71052814"} Sep 15 20:48:14.161777 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.161474 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79ce8f5b3d7ec6cba7a24fc8bc76857d5d3635948743cc1afb3a466b71052814" Sep 15 20:48:14.163462 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.163435 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" Sep 15 20:48:14.163611 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.163437 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvdldd" event={"ID":"c840834d-eb50-4d56-868e-2da7582e9f0a","Type":"ContainerDied","Data":"a1e64d32c1147033746b8ff80279edebc990bd5384d0d279955c6cf2b89c3925"} Sep 15 20:48:14.163611 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.163553 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e64d32c1147033746b8ff80279edebc990bd5384d0d279955c6cf2b89c3925" Sep 15 20:48:14.165623 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.165598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" event={"ID":"121f4680-1852-4e8c-a460-ad9fff98915c","Type":"ContainerDied","Data":"ba58e3aecdae63f64b42ae4e4d6015e85114e670d83ca56b5b7b8013a951ed20"} Sep 15 20:48:14.165721 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.165639 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fjg98" Sep 15 20:48:14.165721 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.165640 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba58e3aecdae63f64b42ae4e4d6015e85114e670d83ca56b5b7b8013a951ed20" Sep 15 20:48:14.284167 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.284141 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:14.353928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.353898 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-util\") pod \"6db47983-124e-476c-bfe9-c29c849c9732\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " Sep 15 20:48:14.354087 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.353935 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-bundle\") pod \"6db47983-124e-476c-bfe9-c29c849c9732\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " Sep 15 20:48:14.354087 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.354009 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg2tb\" (UniqueName: \"kubernetes.io/projected/6db47983-124e-476c-bfe9-c29c849c9732-kube-api-access-vg2tb\") pod \"6db47983-124e-476c-bfe9-c29c849c9732\" (UID: \"6db47983-124e-476c-bfe9-c29c849c9732\") " Sep 15 20:48:14.354623 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.354591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-bundle" (OuterVolumeSpecName: "bundle") pod "6db47983-124e-476c-bfe9-c29c849c9732" (UID: "6db47983-124e-476c-bfe9-c29c849c9732"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:14.356034 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.356010 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db47983-124e-476c-bfe9-c29c849c9732-kube-api-access-vg2tb" (OuterVolumeSpecName: "kube-api-access-vg2tb") pod "6db47983-124e-476c-bfe9-c29c849c9732" (UID: "6db47983-124e-476c-bfe9-c29c849c9732"). InnerVolumeSpecName "kube-api-access-vg2tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:48:14.361438 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.361387 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-util" (OuterVolumeSpecName: "util") pod "6db47983-124e-476c-bfe9-c29c849c9732" (UID: "6db47983-124e-476c-bfe9-c29c849c9732"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:14.455360 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.455293 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg2tb\" (UniqueName: \"kubernetes.io/projected/6db47983-124e-476c-bfe9-c29c849c9732-kube-api-access-vg2tb\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:14.455360 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.455318 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:14.455360 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:14.455328 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6db47983-124e-476c-bfe9-c29c849c9732-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:15.173011 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:15.172978 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" Sep 15 20:48:15.173383 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:15.173009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asrfjx" event={"ID":"6db47983-124e-476c-bfe9-c29c849c9732","Type":"ContainerDied","Data":"5d890627c94ccb3cd7b4316d903e7b4c3c8c216b2df364d72bbb56b709a55adf"} Sep 15 20:48:15.173383 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:15.173058 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d890627c94ccb3cd7b4316d903e7b4c3c8c216b2df364d72bbb56b709a55adf" Sep 15 20:48:17.537968 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:17.537938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:18.322251 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:18.322220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:21.535468 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:21.535435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:21.823271 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:21.823171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:23.525082 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:23.525050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:23.776144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:23.776023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:23.937226 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:23.937193 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:24.528630 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:24.528597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:24.808522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:24.808437 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:24.894544 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:24.894510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:27.145919 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.145883 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w"] Sep 15 20:48:27.146286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146235 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db47983-124e-476c-bfe9-c29c849c9732" containerName="util" Sep 15 20:48:27.146286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146246 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db47983-124e-476c-bfe9-c29c849c9732" containerName="util" Sep 15 20:48:27.146286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146256 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121f4680-1852-4e8c-a460-ad9fff98915c" containerName="pull" Sep 15 20:48:27.146286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146261 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="121f4680-1852-4e8c-a460-ad9fff98915c" containerName="pull" Sep 15 20:48:27.146286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146275 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db47983-124e-476c-bfe9-c29c849c9732" containerName="extract" Sep 15 20:48:27.146286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db47983-124e-476c-bfe9-c29c849c9732" containerName="extract" Sep 15 20:48:27.146286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146289 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121f4680-1852-4e8c-a460-ad9fff98915c" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146294 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="121f4680-1852-4e8c-a460-ad9fff98915c" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146303 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerName="pull" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146308 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerName="pull" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146315 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121f4680-1852-4e8c-a460-ad9fff98915c" containerName="util" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146320 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="121f4680-1852-4e8c-a460-ad9fff98915c" containerName="util" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146325 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerName="util" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146330 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerName="util" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146335 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146341 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146349 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerName="util" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146354 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerName="util" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146359 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerName="pull" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146365 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerName="pull" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146373 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146379 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146385 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db47983-124e-476c-bfe9-c29c849c9732" containerName="pull" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146390 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db47983-124e-476c-bfe9-c29c849c9732" containerName="pull" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146465 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddba387-5f4f-46db-99b1-17dfb64ab6d3" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146476 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="121f4680-1852-4e8c-a460-ad9fff98915c" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146482 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db47983-124e-476c-bfe9-c29c849c9732" containerName="extract" Sep 15 20:48:27.146541 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.146489 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c840834d-eb50-4d56-868e-2da7582e9f0a" containerName="extract" Sep 15 20:48:27.149796 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.149773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.153673 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.153655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 15 20:48:27.153898 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.153883 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rp6ch\"" Sep 15 20:48:27.154055 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.154043 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 15 20:48:27.163688 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.163667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w"] Sep 15 20:48:27.260701 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.260673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzs2h\" (UniqueName: \"kubernetes.io/projected/2ff13d72-6a5d-4534-9a29-04c67b22d36f-kube-api-access-lzs2h\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.260852 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.260783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff13d72-6a5d-4534-9a29-04c67b22d36f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.362167 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.362132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzs2h\" (UniqueName: \"kubernetes.io/projected/2ff13d72-6a5d-4534-9a29-04c67b22d36f-kube-api-access-lzs2h\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.362315 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.362214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff13d72-6a5d-4534-9a29-04c67b22d36f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.362572 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.362552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff13d72-6a5d-4534-9a29-04c67b22d36f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.370276 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.370249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzs2h\" (UniqueName: \"kubernetes.io/projected/2ff13d72-6a5d-4534-9a29-04c67b22d36f-kube-api-access-lzs2h\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.460657 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.460589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:27.747130 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.747105 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w"] Sep 15 20:48:27.748822 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:48:27.748786 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff13d72_6a5d_4534_9a29_04c67b22d36f.slice/crio-e00f5232d560ac404984d2eb583c328c82a62fb789cabef507294dd991643c93 WatchSource:0}: Error finding container e00f5232d560ac404984d2eb583c328c82a62fb789cabef507294dd991643c93: Status 404 returned error can't find the container with id e00f5232d560ac404984d2eb583c328c82a62fb789cabef507294dd991643c93 Sep 15 20:48:27.749455 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:27.749428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:28.244043 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:28.244004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" event={"ID":"2ff13d72-6a5d-4534-9a29-04c67b22d36f","Type":"ContainerStarted","Data":"e00f5232d560ac404984d2eb583c328c82a62fb789cabef507294dd991643c93"} Sep 15 20:48:29.173860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:29.173824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:30.733238 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:30.733199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:30.834696 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:30.834653 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:31.132555 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:31.132516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:31.188995 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:31.188956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:33.279378 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:33.279338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" event={"ID":"2ff13d72-6a5d-4534-9a29-04c67b22d36f","Type":"ContainerStarted","Data":"de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9"} Sep 15 20:48:33.279802 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:33.279443 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:33.304764 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:33.304718 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" podStartSLOduration=1.506060648 podStartE2EDuration="6.304700646s" totalImagesPullingTime="4.798639998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:33.303522281 +0000 UTC m=+702.433258864" watchObservedRunningTime="2026-09-15 20:48:33.304700646 +0000 UTC m=+702.434437231" Sep 15 20:48:34.735056 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:34.735016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:34.906969 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:34.906935 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:35.238611 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.238578 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5"] Sep 15 20:48:35.242191 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.242175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.244663 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.244634 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5k97\"" Sep 15 20:48:35.244972 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.244957 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 15 20:48:35.245069 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.245048 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 15 20:48:35.247120 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.247093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:35.252303 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.252277 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5"] Sep 15 20:48:35.336657 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.336626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fc39bbf-541f-4898-921c-333b939a1106-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.336820 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.336719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fc39bbf-541f-4898-921c-333b939a1106-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.336820 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.336743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hlj\" (UniqueName: \"kubernetes.io/projected/3fc39bbf-541f-4898-921c-333b939a1106-kube-api-access-56hlj\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.437874 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.437846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fc39bbf-541f-4898-921c-333b939a1106-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.438044 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.437880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hlj\" (UniqueName: \"kubernetes.io/projected/3fc39bbf-541f-4898-921c-333b939a1106-kube-api-access-56hlj\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.438044 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.437929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fc39bbf-541f-4898-921c-333b939a1106-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.438044 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:48:35.438015 2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 15 20:48:35.438182 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:48:35.438099 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc39bbf-541f-4898-921c-333b939a1106-plugin-serving-cert podName:3fc39bbf-541f-4898-921c-333b939a1106 nodeName:}" failed. No retries permitted until 2026-09-15 20:48:35.938077641 +0000 UTC m=+705.067814208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3fc39bbf-541f-4898-921c-333b939a1106-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-p74c5" (UID: "3fc39bbf-541f-4898-921c-333b939a1106") : secret "plugin-serving-cert" not found Sep 15 20:48:35.438705 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.438687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fc39bbf-541f-4898-921c-333b939a1106-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.455648 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.455624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hlj\" (UniqueName: \"kubernetes.io/projected/3fc39bbf-541f-4898-921c-333b939a1106-kube-api-access-56hlj\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.625599 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.625562 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:35.943364 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.943275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fc39bbf-541f-4898-921c-333b939a1106-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:35.945721 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:35.945694 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fc39bbf-541f-4898-921c-333b939a1106-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-p74c5\" (UID: \"3fc39bbf-541f-4898-921c-333b939a1106\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:36.153000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:36.152965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" Sep 15 20:48:36.423702 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:36.423674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5"] Sep 15 20:48:37.304535 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:37.304499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" event={"ID":"3fc39bbf-541f-4898-921c-333b939a1106","Type":"ContainerStarted","Data":"e3c21eee8a3bc9cd67872b81a1eebebacd1ade5c9549e85ad761daab1488ca64"} Sep 15 20:48:38.537116 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:38.537078 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:38.637251 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:38.637219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:38.764349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:38.764310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:42.336833 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:42.336793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" event={"ID":"3fc39bbf-541f-4898-921c-333b939a1106","Type":"ContainerStarted","Data":"44914170cc172cd108e9621d8f014d4baae6a4827a0eec473ec62f1397337980"} Sep 15 20:48:42.356687 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:42.356639 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-p74c5" podStartSLOduration=2.033612103 podStartE2EDuration="7.356624474s" totalImagesPullingTime="5.323012371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:42.355513394 +0000 UTC m=+711.485249978" watchObservedRunningTime="2026-09-15 20:48:42.356624474 +0000 UTC m=+711.486361058" Sep 15 20:48:44.286162 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:44.286083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:45.146599 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:45.146563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:45.278394 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:45.278358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:46.720780 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:46.720749 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w"] Sep 15 20:48:46.721171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:46.720986 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" podUID="2ff13d72-6a5d-4534-9a29-04c67b22d36f" containerName="manager" containerID="cri-o://de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9" gracePeriod=2 Sep 15 20:48:46.731994 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:46.731963 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w"] Sep 15 20:48:46.733966 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:48:46.733935 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2ff13d72-6a5d-4534-9a29-04c67b22d36f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-wf27w\": pods \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:46.956589 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:46.956565 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:46.958334 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:48:46.958307 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-wf27w\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="2ff13d72-6a5d-4534-9a29-04c67b22d36f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:47.049425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.049376 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzs2h\" (UniqueName: \"kubernetes.io/projected/2ff13d72-6a5d-4534-9a29-04c67b22d36f-kube-api-access-lzs2h\") pod \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " Sep 15 20:48:47.049557 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.049496 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff13d72-6a5d-4534-9a29-04c67b22d36f-extensions-socket-volume\") pod \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\" (UID: \"2ff13d72-6a5d-4534-9a29-04c67b22d36f\") " Sep 15 20:48:47.049991 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.049967 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff13d72-6a5d-4534-9a29-04c67b22d36f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2ff13d72-6a5d-4534-9a29-04c67b22d36f" (UID: "2ff13d72-6a5d-4534-9a29-04c67b22d36f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:48:47.051622 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.051601 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff13d72-6a5d-4534-9a29-04c67b22d36f-kube-api-access-lzs2h" (OuterVolumeSpecName: "kube-api-access-lzs2h") pod "2ff13d72-6a5d-4534-9a29-04c67b22d36f" (UID: "2ff13d72-6a5d-4534-9a29-04c67b22d36f"). InnerVolumeSpecName "kube-api-access-lzs2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:48:47.150290 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.150249 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff13d72-6a5d-4534-9a29-04c67b22d36f-extensions-socket-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:47.150290 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.150286 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzs2h\" (UniqueName: \"kubernetes.io/projected/2ff13d72-6a5d-4534-9a29-04c67b22d36f-kube-api-access-lzs2h\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:48:47.314235 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.314151 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff13d72-6a5d-4534-9a29-04c67b22d36f" path="/var/lib/kubelet/pods/2ff13d72-6a5d-4534-9a29-04c67b22d36f/volumes" Sep 15 20:48:47.366124 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.366094 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ff13d72-6a5d-4534-9a29-04c67b22d36f" containerID="de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9" exitCode=0 Sep 15 20:48:47.366292 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.366141 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf27w" Sep 15 20:48:47.366292 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.366165 2580 scope.go:117] "RemoveContainer" containerID="de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9" Sep 15 20:48:47.375143 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.375127 2580 scope.go:117] "RemoveContainer" containerID="de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9" Sep 15 20:48:47.375447 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:48:47.375427 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9\": container with ID starting with de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9 not found: ID does not exist" containerID="de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9" Sep 15 20:48:47.375533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:47.375456 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9"} err="failed to get container status \"de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9\": rpc error: code = NotFound desc = could not find container \"de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9\": container with ID starting with de3a983f5d9fae2d29d2b81f423eab502b045e1dd954e825507c0634b5ff59a9 not found: ID does not exist" Sep 15 20:48:50.775468 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:50.775437 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:50.831100 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:50.831067 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:51.332492 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:51.332462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:51.391115 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:51.391086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:54.736706 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:54.736665 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:54.842299 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:54.842262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:55.201695 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:55.201653 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:58.726823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:58.726791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:58.774485 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:58.774455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:58.948294 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:58.948266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:59.006215 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:59.006187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:59.127706 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:59.127672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:48:59.446458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:48:59.446347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:02.305908 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.305869 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns"] Sep 15 20:49:02.306286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.306240 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff13d72-6a5d-4534-9a29-04c67b22d36f" containerName="manager" Sep 15 20:49:02.306286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.306264 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff13d72-6a5d-4534-9a29-04c67b22d36f" containerName="manager" Sep 15 20:49:02.306372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.306339 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff13d72-6a5d-4534-9a29-04c67b22d36f" containerName="manager" Sep 15 20:49:02.314928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.314905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.317077 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.317058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rp6ch\"" Sep 15 20:49:02.324763 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.324736 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns"] Sep 15 20:49:02.374204 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.374179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cf11874-73cc-4d79-9d9e-afbe42301acf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pxrns\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.374320 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.374223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtdqf\" (UniqueName: \"kubernetes.io/projected/2cf11874-73cc-4d79-9d9e-afbe42301acf-kube-api-access-mtdqf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pxrns\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.475020 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.474988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cf11874-73cc-4d79-9d9e-afbe42301acf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pxrns\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.475147 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.475032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtdqf\" (UniqueName: \"kubernetes.io/projected/2cf11874-73cc-4d79-9d9e-afbe42301acf-kube-api-access-mtdqf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pxrns\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.475373 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.475353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cf11874-73cc-4d79-9d9e-afbe42301acf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pxrns\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.486632 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.486602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtdqf\" (UniqueName: \"kubernetes.io/projected/2cf11874-73cc-4d79-9d9e-afbe42301acf-kube-api-access-mtdqf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pxrns\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.626469 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.626379 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:02.920896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:02.920785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns"] Sep 15 20:49:02.923656 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:49:02.923624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf11874_73cc_4d79_9d9e_afbe42301acf.slice/crio-7fcf2e2c17f635f73b91d90206b9c518e1f7adf04f1421de65683b677537b4a4 WatchSource:0}: Error finding container 7fcf2e2c17f635f73b91d90206b9c518e1f7adf04f1421de65683b677537b4a4: Status 404 returned error can't find the container with id 7fcf2e2c17f635f73b91d90206b9c518e1f7adf04f1421de65683b677537b4a4 Sep 15 20:49:03.452375 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:03.452332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" event={"ID":"2cf11874-73cc-4d79-9d9e-afbe42301acf","Type":"ContainerStarted","Data":"031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3"} Sep 15 20:49:03.452809 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:03.452396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" event={"ID":"2cf11874-73cc-4d79-9d9e-afbe42301acf","Type":"ContainerStarted","Data":"7fcf2e2c17f635f73b91d90206b9c518e1f7adf04f1421de65683b677537b4a4"} Sep 15 20:49:03.452809 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:03.452581 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:03.478500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:03.478453 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" podStartSLOduration=1.478438327 podStartE2EDuration="1.478438327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:49:03.476296462 +0000 UTC m=+732.606033046" watchObservedRunningTime="2026-09-15 20:49:03.478438327 +0000 UTC m=+732.608174910" Sep 15 20:49:05.043124 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:05.043086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:05.143207 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:05.143175 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:14.460173 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:14.460141 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 20:49:18.562194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.562157 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw"] Sep 15 20:49:18.567546 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.567509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.571193 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.571168 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w4n8g\"" Sep 15 20:49:18.575961 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.575934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw"] Sep 15 20:49:18.581040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.581008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:18.719860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.719827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.719870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.719893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.719910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56sxz\" (UniqueName: \"kubernetes.io/projected/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-kube-api-access-56sxz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.719992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720040 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.720029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.720113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.720133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.720260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.720147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.764913 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.764865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:18.821064 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.820994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56sxz\" (UniqueName: \"kubernetes.io/projected/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-kube-api-access-56sxz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821064 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821064 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821720 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.821890 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.821770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.822027 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.822005 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.823358 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.823337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.823676 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.823659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.829057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.829032 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.829269 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.829250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56sxz\" (UniqueName: \"kubernetes.io/projected/2e53a04c-0eeb-4627-962b-a4fbb8dff2fe-kube-api-access-56sxz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kwffw\" (UID: \"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:18.881280 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:18.881254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:19.157033 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.157006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw"] Sep 15 20:49:19.159205 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:49:19.159178 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e53a04c_0eeb_4627_962b_a4fbb8dff2fe.slice/crio-fab43d3ab2f8df36c8c9e274fc733ea2fd2e012eb5288e995af261a07502f9b6 WatchSource:0}: Error finding container fab43d3ab2f8df36c8c9e274fc733ea2fd2e012eb5288e995af261a07502f9b6: Status 404 returned error can't find the container with id fab43d3ab2f8df36c8c9e274fc733ea2fd2e012eb5288e995af261a07502f9b6 Sep 15 20:49:19.161353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.161319 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:49:19.161467 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.161383 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:49:19.161467 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.161424 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:49:19.542926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.542898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" event={"ID":"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe","Type":"ContainerStarted","Data":"db992fadc97de5f94d5e53cf2c312950d40ad81d1876da7698f5fd180705cf41"} Sep 15 20:49:19.543101 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.542951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" event={"ID":"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe","Type":"ContainerStarted","Data":"fab43d3ab2f8df36c8c9e274fc733ea2fd2e012eb5288e995af261a07502f9b6"} Sep 15 20:49:19.564430 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.562373 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" podStartSLOduration=1.5623521870000001 podStartE2EDuration="1.562352187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:49:19.560339549 +0000 UTC m=+748.690076132" watchObservedRunningTime="2026-09-15 20:49:19.562352187 +0000 UTC m=+748.692088781" Sep 15 20:49:19.881875 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:19.881802 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:20.887091 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:20.887061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:21.553369 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:21.553342 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:21.554312 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:21.554289 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:49:30.751618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:30.751583 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:30.854560 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:30.854525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:30.956165 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:30.956127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:31.642315 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:31.642280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:31.742204 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:31.742157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:31.811947 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:31.811917 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g76v"] Sep 15 20:49:31.815799 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:31.815777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g76v" Sep 15 20:49:31.817930 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:31.817909 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6kdkm\"" Sep 15 20:49:31.821526 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:31.821504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g76v"] Sep 15 20:49:31.932841 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:31.932768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb576\" (UniqueName: \"kubernetes.io/projected/a08f9a8e-b02d-4174-8e52-e7a6463a1525-kube-api-access-sb576\") pod \"authorino-59c974bd48-9g76v\" (UID: \"a08f9a8e-b02d-4174-8e52-e7a6463a1525\") " pod="kuadrant-system/authorino-59c974bd48-9g76v" Sep 15 20:49:32.033386 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:32.033357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb576\" (UniqueName: \"kubernetes.io/projected/a08f9a8e-b02d-4174-8e52-e7a6463a1525-kube-api-access-sb576\") pod \"authorino-59c974bd48-9g76v\" (UID: \"a08f9a8e-b02d-4174-8e52-e7a6463a1525\") " pod="kuadrant-system/authorino-59c974bd48-9g76v" Sep 15 20:49:32.041279 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:32.041254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb576\" (UniqueName: \"kubernetes.io/projected/a08f9a8e-b02d-4174-8e52-e7a6463a1525-kube-api-access-sb576\") pod \"authorino-59c974bd48-9g76v\" (UID: \"a08f9a8e-b02d-4174-8e52-e7a6463a1525\") " pod="kuadrant-system/authorino-59c974bd48-9g76v" Sep 15 20:49:32.126927 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:32.126897 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g76v" Sep 15 20:49:32.397987 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:32.397964 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g76v"] Sep 15 20:49:32.399881 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:49:32.399854 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda08f9a8e_b02d_4174_8e52_e7a6463a1525.slice/crio-2eb7ffcaa7860c0690ad5dedf07f357d18a9ebf20ed5ac0a9a4b4c6064e5128e WatchSource:0}: Error finding container 2eb7ffcaa7860c0690ad5dedf07f357d18a9ebf20ed5ac0a9a4b4c6064e5128e: Status 404 returned error can't find the container with id 2eb7ffcaa7860c0690ad5dedf07f357d18a9ebf20ed5ac0a9a4b4c6064e5128e Sep 15 20:49:32.613643 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:32.613609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g76v" event={"ID":"a08f9a8e-b02d-4174-8e52-e7a6463a1525","Type":"ContainerStarted","Data":"2eb7ffcaa7860c0690ad5dedf07f357d18a9ebf20ed5ac0a9a4b4c6064e5128e"} Sep 15 20:49:34.627127 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:34.627085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g76v" event={"ID":"a08f9a8e-b02d-4174-8e52-e7a6463a1525","Type":"ContainerStarted","Data":"8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83"} Sep 15 20:49:34.640816 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:34.640767 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-9g76v" podStartSLOduration=2.03440426 podStartE2EDuration="3.640752026s" totalImagesPullingTime="1.606347766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:49:34.640207486 +0000 UTC m=+763.769944070" watchObservedRunningTime="2026-09-15 20:49:34.640752026 +0000 UTC m=+763.770488609" Sep 15 20:49:47.170919 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:47.170868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:47.270273 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:47.270229 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:49:47.693597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:49:47.693564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:00.688153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:00.688119 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:00.902270 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:00.902239 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:01.195553 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.195517 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dc4dc659-6c5n5"] Sep 15 20:50:01.201636 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.201610 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.203718 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.203695 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 15 20:50:01.204985 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.204964 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dc4dc659-6c5n5"] Sep 15 20:50:01.276824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.276795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ddbd996-d756-4cf4-93ba-df00ecbd2345-tls-cert\") pod \"authorino-6dc4dc659-6c5n5\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.276942 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.276846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hkc\" (UniqueName: \"kubernetes.io/projected/4ddbd996-d756-4cf4-93ba-df00ecbd2345-kube-api-access-j9hkc\") pod \"authorino-6dc4dc659-6c5n5\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.377535 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.377508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hkc\" (UniqueName: \"kubernetes.io/projected/4ddbd996-d756-4cf4-93ba-df00ecbd2345-kube-api-access-j9hkc\") pod \"authorino-6dc4dc659-6c5n5\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.377679 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.377589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ddbd996-d756-4cf4-93ba-df00ecbd2345-tls-cert\") pod \"authorino-6dc4dc659-6c5n5\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.379956 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.379930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ddbd996-d756-4cf4-93ba-df00ecbd2345-tls-cert\") pod \"authorino-6dc4dc659-6c5n5\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.384094 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.384068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hkc\" (UniqueName: \"kubernetes.io/projected/4ddbd996-d756-4cf4-93ba-df00ecbd2345-kube-api-access-j9hkc\") pod \"authorino-6dc4dc659-6c5n5\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.512157 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.512125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:50:01.779973 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:01.779946 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dc4dc659-6c5n5"] Sep 15 20:50:01.781866 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:50:01.781836 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ddbd996_d756_4cf4_93ba_df00ecbd2345.slice/crio-640453cedab1f58d2f19e002395ffbb7ddb19ac32a1da3746276177852bc9910 WatchSource:0}: Error finding container 640453cedab1f58d2f19e002395ffbb7ddb19ac32a1da3746276177852bc9910: Status 404 returned error can't find the container with id 640453cedab1f58d2f19e002395ffbb7ddb19ac32a1da3746276177852bc9910 Sep 15 20:50:02.785948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:02.785917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" event={"ID":"4ddbd996-d756-4cf4-93ba-df00ecbd2345","Type":"ContainerStarted","Data":"92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784"} Sep 15 20:50:02.786374 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:02.785960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" event={"ID":"4ddbd996-d756-4cf4-93ba-df00ecbd2345","Type":"ContainerStarted","Data":"640453cedab1f58d2f19e002395ffbb7ddb19ac32a1da3746276177852bc9910"} Sep 15 20:50:02.799818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:02.799770 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" podStartSLOduration=0.951178993 podStartE2EDuration="1.799752209s" totalImagesPullingTime="848.573216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:50:02.798790879 +0000 UTC m=+791.928527465" watchObservedRunningTime="2026-09-15 20:50:02.799752209 +0000 UTC m=+791.929488791" Sep 15 20:50:02.825849 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:02.825817 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g76v"] Sep 15 20:50:02.826220 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:02.826184 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-9g76v" podUID="a08f9a8e-b02d-4174-8e52-e7a6463a1525" containerName="authorino" containerID="cri-o://8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83" gracePeriod=30 Sep 15 20:50:03.070278 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.070254 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g76v" Sep 15 20:50:03.191594 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.191559 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb576\" (UniqueName: \"kubernetes.io/projected/a08f9a8e-b02d-4174-8e52-e7a6463a1525-kube-api-access-sb576\") pod \"a08f9a8e-b02d-4174-8e52-e7a6463a1525\" (UID: \"a08f9a8e-b02d-4174-8e52-e7a6463a1525\") " Sep 15 20:50:03.193837 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.193808 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08f9a8e-b02d-4174-8e52-e7a6463a1525-kube-api-access-sb576" (OuterVolumeSpecName: "kube-api-access-sb576") pod "a08f9a8e-b02d-4174-8e52-e7a6463a1525" (UID: "a08f9a8e-b02d-4174-8e52-e7a6463a1525"). InnerVolumeSpecName "kube-api-access-sb576". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:50:03.292649 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.292616 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb576\" (UniqueName: \"kubernetes.io/projected/a08f9a8e-b02d-4174-8e52-e7a6463a1525-kube-api-access-sb576\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:50:03.792750 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.792716 2580 generic.go:356] "Generic (PLEG): container finished" podID="a08f9a8e-b02d-4174-8e52-e7a6463a1525" containerID="8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83" exitCode=0 Sep 15 20:50:03.793129 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.792758 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9g76v" Sep 15 20:50:03.793129 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.792795 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g76v" event={"ID":"a08f9a8e-b02d-4174-8e52-e7a6463a1525","Type":"ContainerDied","Data":"8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83"} Sep 15 20:50:03.793129 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.792846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9g76v" event={"ID":"a08f9a8e-b02d-4174-8e52-e7a6463a1525","Type":"ContainerDied","Data":"2eb7ffcaa7860c0690ad5dedf07f357d18a9ebf20ed5ac0a9a4b4c6064e5128e"} Sep 15 20:50:03.793129 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.792871 2580 scope.go:117] "RemoveContainer" containerID="8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83" Sep 15 20:50:03.801530 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.801515 2580 scope.go:117] "RemoveContainer" containerID="8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83" Sep 15 20:50:03.801795 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:50:03.801777 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83\": container with ID starting with 8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83 not found: ID does not exist" containerID="8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83" Sep 15 20:50:03.801852 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.801804 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83"} err="failed to get container status \"8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83\": rpc error: code = NotFound desc = could not find container \"8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83\": container with ID starting with 8c4bc6823bdbe0f2ef5ec525f431d8857cfebfefae0772c4c8f009ea79ee7c83 not found: ID does not exist" Sep 15 20:50:03.808029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.807984 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g76v"] Sep 15 20:50:03.811263 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:03.811239 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9g76v"] Sep 15 20:50:04.717658 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:04.717627 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:04.818913 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:04.818878 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:05.313666 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:05.313636 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a08f9a8e-b02d-4174-8e52-e7a6463a1525" path="/var/lib/kubelet/pods/a08f9a8e-b02d-4174-8e52-e7a6463a1525/volumes" Sep 15 20:50:07.749695 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:07.749650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:07.843390 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:07.843348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:07.968859 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:07.968827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:08.148031 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:08.147995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:11.322870 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:11.322834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:11.422083 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:11.422050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:42.729389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:42.729301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:42.826474 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:42.826398 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:44.123173 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.123133 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:44.224702 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.224667 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:44.341353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.341321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:44.560467 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.560434 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:44.911975 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.911896 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw"] Sep 15 20:50:44.912580 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.912560 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a08f9a8e-b02d-4174-8e52-e7a6463a1525" containerName="authorino" Sep 15 20:50:44.912663 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.912588 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a08f9a8e-b02d-4174-8e52-e7a6463a1525" containerName="authorino" Sep 15 20:50:44.912735 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.912723 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a08f9a8e-b02d-4174-8e52-e7a6463a1525" containerName="authorino" Sep 15 20:50:44.916440 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.916419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:44.919460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.919427 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 15 20:50:44.919460 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.919442 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l5l9x\"" Sep 15 20:50:44.921708 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:44.921686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw"] Sep 15 20:50:45.039908 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.039876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e44d4f-614f-4962-8021-1b5dbc4edc74-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lqfgw\" (UID: \"47e44d4f-614f-4962-8021-1b5dbc4edc74\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:45.039908 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.039920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfj7\" (UniqueName: \"kubernetes.io/projected/47e44d4f-614f-4962-8021-1b5dbc4edc74-kube-api-access-xkfj7\") pod \"payload-pre-processing-59fbc47bf8-lqfgw\" (UID: \"47e44d4f-614f-4962-8021-1b5dbc4edc74\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:45.140722 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.140686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfj7\" (UniqueName: \"kubernetes.io/projected/47e44d4f-614f-4962-8021-1b5dbc4edc74-kube-api-access-xkfj7\") pod \"payload-pre-processing-59fbc47bf8-lqfgw\" (UID: \"47e44d4f-614f-4962-8021-1b5dbc4edc74\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:45.141204 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.140800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e44d4f-614f-4962-8021-1b5dbc4edc74-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lqfgw\" (UID: \"47e44d4f-614f-4962-8021-1b5dbc4edc74\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:45.141467 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.141446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e44d4f-614f-4962-8021-1b5dbc4edc74-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lqfgw\" (UID: \"47e44d4f-614f-4962-8021-1b5dbc4edc74\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:45.149908 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.149881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfj7\" (UniqueName: \"kubernetes.io/projected/47e44d4f-614f-4962-8021-1b5dbc4edc74-kube-api-access-xkfj7\") pod \"payload-pre-processing-59fbc47bf8-lqfgw\" (UID: \"47e44d4f-614f-4962-8021-1b5dbc4edc74\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:45.229787 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.229705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:45.301220 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.301129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:45.400910 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.400880 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:45.505745 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:45.505719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw"] Sep 15 20:50:45.507494 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:50:45.507466 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e44d4f_614f_4962_8021_1b5dbc4edc74.slice/crio-2adf0fc8eea1bc5bd9627ce56c750c04dc82af4f571bcabe2d01274ac57ec24d WatchSource:0}: Error finding container 2adf0fc8eea1bc5bd9627ce56c750c04dc82af4f571bcabe2d01274ac57ec24d: Status 404 returned error can't find the container with id 2adf0fc8eea1bc5bd9627ce56c750c04dc82af4f571bcabe2d01274ac57ec24d Sep 15 20:50:46.035391 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:46.035345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" event={"ID":"47e44d4f-614f-4962-8021-1b5dbc4edc74","Type":"ContainerStarted","Data":"2adf0fc8eea1bc5bd9627ce56c750c04dc82af4f571bcabe2d01274ac57ec24d"} Sep 15 20:50:48.049366 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:48.049327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" event={"ID":"47e44d4f-614f-4962-8021-1b5dbc4edc74","Type":"ContainerStarted","Data":"f1f4fe1182e13aa48994852df3f5d3dcf488f81165d25429e3efc43bccf74868"} Sep 15 20:50:48.049871 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:48.049436 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:48.066185 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:48.066135 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" podStartSLOduration=2.592124719 podStartE2EDuration="4.066120214s" totalImagesPullingTime="1.473995495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:50:48.062492891 +0000 UTC m=+837.192229473" watchObservedRunningTime="2026-09-15 20:50:48.066120214 +0000 UTC m=+837.195856797" Sep 15 20:50:52.594561 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:52.594520 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kwffw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 15 20:50:52.594951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:52.594609 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" podUID="2e53a04c-0eeb-4627-962b-a4fbb8dff2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 15 20:50:54.085615 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.085537 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e53a04c-0eeb-4627-962b-a4fbb8dff2fe" containerID="db992fadc97de5f94d5e53cf2c312950d40ad81d1876da7698f5fd180705cf41" exitCode=0 Sep 15 20:50:54.086045 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.085610 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" event={"ID":"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe","Type":"ContainerDied","Data":"db992fadc97de5f94d5e53cf2c312950d40ad81d1876da7698f5fd180705cf41"} Sep 15 20:50:54.086045 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.085966 2580 scope.go:117] "RemoveContainer" containerID="db992fadc97de5f94d5e53cf2c312950d40ad81d1876da7698f5fd180705cf41" Sep 15 20:50:54.086990 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.086963 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:50:54.087048 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.087031 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:50:54.087086 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.087063 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:50:54.857335 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.857294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:54.958863 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:54.958826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:50:55.093559 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:55.093526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" event={"ID":"2e53a04c-0eeb-4627-962b-a4fbb8dff2fe","Type":"ContainerStarted","Data":"86a0889e95835d32ba014507544bcb0bc9e35e70a308cf6ac1711d18397b3da2"} Sep 15 20:50:55.881610 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:55.881574 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:50:55.881791 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:55.881773 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kwffw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:50:55.881840 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:55.881817 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" podUID="2e53a04c-0eeb-4627-962b-a4fbb8dff2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 15 20:50:56.881878 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:56.881836 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kwffw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:50:56.882352 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:56.881899 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" podUID="2e53a04c-0eeb-4627-962b-a4fbb8dff2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 15 20:50:57.882593 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:57.882556 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kwffw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:50:57.882988 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:57.882611 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" podUID="2e53a04c-0eeb-4627-962b-a4fbb8dff2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 15 20:50:58.882275 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:58.882235 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kwffw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:50:58.882508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:58.882302 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" podUID="2e53a04c-0eeb-4627-962b-a4fbb8dff2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 15 20:50:59.058245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:59.058214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lqfgw" Sep 15 20:50:59.881728 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:59.881692 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kwffw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 15 20:50:59.881912 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:50:59.881749 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" podUID="2e53a04c-0eeb-4627-962b-a4fbb8dff2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 15 20:51:00.294091 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:00.294057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:51:00.885101 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:00.885064 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:51:00.885457 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:00.885435 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:51:00.886131 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:00.886112 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kwffw" Sep 15 20:51:13.485196 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:13.485163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:51:23.362670 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.362634 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4"] Sep 15 20:51:23.367145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.367119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.370057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.370015 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 20:51:23.370190 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.370087 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bwfmg\"" Sep 15 20:51:23.370190 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.370029 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 20:51:23.370312 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.370296 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 15 20:51:23.373697 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.373666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4"] Sep 15 20:51:23.391559 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.391527 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:51:23.393639 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.393604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.393639 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.393631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.393815 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.393677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zkt9\" (UniqueName: \"kubernetes.io/projected/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-kube-api-access-8zkt9\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.393815 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.393768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.393815 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.393795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.393939 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.393888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495226 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495367 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495367 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495367 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkt9\" (UniqueName: \"kubernetes.io/projected/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-kube-api-access-8zkt9\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495584 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495584 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495700 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495756 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.495805 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.495774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.497549 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.497528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.497821 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.497803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.502071 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.502045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zkt9\" (UniqueName: \"kubernetes.io/projected/06ba2a11-64d6-4e25-b35c-82b3b7a1f000-kube-api-access-8zkt9\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4\" (UID: \"06ba2a11-64d6-4e25-b35c-82b3b7a1f000\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.679080 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.678992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:23.955184 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:23.955154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4"] Sep 15 20:51:23.956474 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:51:23.956440 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ba2a11_64d6_4e25_b35c_82b3b7a1f000.slice/crio-05408ecc5c49f78b90525333e34dfe0fdfc57e319fe0b7a50cae4a9740bfac79 WatchSource:0}: Error finding container 05408ecc5c49f78b90525333e34dfe0fdfc57e319fe0b7a50cae4a9740bfac79: Status 404 returned error can't find the container with id 05408ecc5c49f78b90525333e34dfe0fdfc57e319fe0b7a50cae4a9740bfac79 Sep 15 20:51:24.267758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:24.267712 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" event={"ID":"06ba2a11-64d6-4e25-b35c-82b3b7a1f000","Type":"ContainerStarted","Data":"05408ecc5c49f78b90525333e34dfe0fdfc57e319fe0b7a50cae4a9740bfac79"} Sep 15 20:51:25.275121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:25.275071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" event={"ID":"06ba2a11-64d6-4e25-b35c-82b3b7a1f000","Type":"ContainerStarted","Data":"e0c7887c269f8bd94ddf6ce4661c380ad9bfe52b06575f1d1bf3c2888e4cfa05"} Sep 15 20:51:25.292114 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:25.292022 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" podStartSLOduration=1.220232598 podStartE2EDuration="2.292006674s" totalImagesPullingTime="1.071774076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:51:25.289687839 +0000 UTC m=+874.419424428" watchObservedRunningTime="2026-09-15 20:51:25.292006674 +0000 UTC m=+874.421743257" Sep 15 20:51:33.679674 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:33.679627 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:33.680060 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:33.679683 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:33.692582 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:33.692558 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:34.338658 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:34.338629 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4" Sep 15 20:51:39.760227 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:39.760191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:51:46.093609 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:46.093579 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:51:51.299169 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:51.299140 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:51:51.302552 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:51.302531 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:51:53.164313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:51:53.164232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:52:03.994811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:52:03.994764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:52:48.218369 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:52:48.218325 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:52:53.677545 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:52:53.677508 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:15.577755 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:15.577674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:28.805102 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:28.805066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:28.852905 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:28.852870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:49.398721 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.398672 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dc4dc659-6c5n5"] Sep 15 20:53:49.399156 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.398916 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" podUID="4ddbd996-d756-4cf4-93ba-df00ecbd2345" containerName="authorino" containerID="cri-o://92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784" gracePeriod=30 Sep 15 20:53:49.640740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.640716 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:53:49.759506 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.759483 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ddbd996-d756-4cf4-93ba-df00ecbd2345-tls-cert\") pod \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " Sep 15 20:53:49.759680 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.759529 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9hkc\" (UniqueName: \"kubernetes.io/projected/4ddbd996-d756-4cf4-93ba-df00ecbd2345-kube-api-access-j9hkc\") pod \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\" (UID: \"4ddbd996-d756-4cf4-93ba-df00ecbd2345\") " Sep 15 20:53:49.761539 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.761508 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddbd996-d756-4cf4-93ba-df00ecbd2345-kube-api-access-j9hkc" (OuterVolumeSpecName: "kube-api-access-j9hkc") pod "4ddbd996-d756-4cf4-93ba-df00ecbd2345" (UID: "4ddbd996-d756-4cf4-93ba-df00ecbd2345"). InnerVolumeSpecName "kube-api-access-j9hkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:53:49.770053 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.770028 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddbd996-d756-4cf4-93ba-df00ecbd2345-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4ddbd996-d756-4cf4-93ba-df00ecbd2345" (UID: "4ddbd996-d756-4cf4-93ba-df00ecbd2345"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:53:49.860209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.860181 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ddbd996-d756-4cf4-93ba-df00ecbd2345-tls-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:53:49.860209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.860205 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9hkc\" (UniqueName: \"kubernetes.io/projected/4ddbd996-d756-4cf4-93ba-df00ecbd2345-kube-api-access-j9hkc\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:53:49.913091 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:49.913055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:50.012170 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.012089 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:50.106472 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.106437 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ddbd996-d756-4cf4-93ba-df00ecbd2345" containerID="92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784" exitCode=0 Sep 15 20:53:50.106631 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.106474 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" Sep 15 20:53:50.106631 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.106495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" event={"ID":"4ddbd996-d756-4cf4-93ba-df00ecbd2345","Type":"ContainerDied","Data":"92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784"} Sep 15 20:53:50.106631 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.106530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc4dc659-6c5n5" event={"ID":"4ddbd996-d756-4cf4-93ba-df00ecbd2345","Type":"ContainerDied","Data":"640453cedab1f58d2f19e002395ffbb7ddb19ac32a1da3746276177852bc9910"} Sep 15 20:53:50.106631 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.106549 2580 scope.go:117] "RemoveContainer" containerID="92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784" Sep 15 20:53:50.116956 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.116935 2580 scope.go:117] "RemoveContainer" containerID="92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784" Sep 15 20:53:50.117262 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:53:50.117243 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784\": container with ID starting with 92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784 not found: ID does not exist" containerID="92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784" Sep 15 20:53:50.117313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.117270 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784"} err="failed to get container status \"92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784\": rpc error: code = NotFound desc = could not find container \"92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784\": container with ID starting with 92df73f208652c9934bc88e71f13d0f16bd0a9beb33885516978d510457a5784 not found: ID does not exist" Sep 15 20:53:50.129457 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.129432 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dc4dc659-6c5n5"] Sep 15 20:53:50.135678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:50.135656 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dc4dc659-6c5n5"] Sep 15 20:53:51.046784 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:51.046745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:51.132891 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:51.132857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:53:51.314372 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:51.314265 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ddbd996-d756-4cf4-93ba-df00ecbd2345" path="/var/lib/kubelet/pods/4ddbd996-d756-4cf4-93ba-df00ecbd2345/volumes" Sep 15 20:53:56.173973 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:53:56.173938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:08.851421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:08.851381 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:08.952552 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:08.952523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.052615 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.052570 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.193632 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.193547 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5"] Sep 15 20:54:09.194162 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.194131 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddbd996-d756-4cf4-93ba-df00ecbd2345" containerName="authorino" Sep 15 20:54:09.194162 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.194160 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddbd996-d756-4cf4-93ba-df00ecbd2345" containerName="authorino" Sep 15 20:54:09.194383 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.194312 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ddbd996-d756-4cf4-93ba-df00ecbd2345" containerName="authorino" Sep 15 20:54:09.198988 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.198967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.201011 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.200983 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-16bde4-openshift-default-dockercfg-mtrdk\"" Sep 15 20:54:09.208941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.208919 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5"] Sep 15 20:54:09.218488 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.218460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.227482 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-socket\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227599 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-envoy\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227599 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ckt\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-kube-api-access-64ckt\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227703 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc431a76-246a-4136-a3eb-a594eba56ed3-istiod-ca-cert\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227703 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-data\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227703 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-podinfo\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227810 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-token\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227851 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-certs\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.227935 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.227916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-credential-socket\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.259316 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.259285 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk"] Sep 15 20:54:09.273927 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.273896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.276095 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.276048 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d5b5b9-openshift-default-dockercfg-2qc8v\"" Sep 15 20:54:09.276559 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.276533 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk"] Sep 15 20:54:09.300832 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.300792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.328542 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-credential-socket\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.328697 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-envoy\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.328697 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmw8r\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-kube-api-access-jmw8r\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.328814 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-istio-token\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.328814 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-socket\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.328814 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ckt\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-kube-api-access-64ckt\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.328814 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-envoy\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-socket\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.329029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-podinfo\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.328963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-credential-socket\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-certs\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.329255 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc431a76-246a-4136-a3eb-a594eba56ed3-istiod-ca-cert\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329255 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-data\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329255 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-socket\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329255 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-token\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329255 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb07e50-b590-4a0c-8244-c45d83329b75-istiod-ca-cert\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.329255 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-data\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.329890 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-certs\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.329890 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.329318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb07e50-b590-4a0c-8244-c45d83329b75-istio-podinfo\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.330132 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.330105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-data\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.330293 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.330140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-certs\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.330492 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.330161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc431a76-246a-4136-a3eb-a594eba56ed3-istiod-ca-cert\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.330492 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.330488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-credential-socket\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.332611 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.332588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-envoy\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.333899 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.333844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-podinfo\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.334916 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.334894 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm"] Sep 15 20:54:09.338573 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.338549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ckt\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-kube-api-access-64ckt\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.338668 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.338611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-token\") pod \"e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.340324 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.340310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.342269 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.342249 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3f1c93-openshift-default-dockercfg-thmcp\"" Sep 15 20:54:09.350099 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.350074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm"] Sep 15 20:54:09.381175 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.381146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.430769 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.430732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp85p\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-kube-api-access-sp85p\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.430921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.430793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb07e50-b590-4a0c-8244-c45d83329b75-istiod-ca-cert\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.430921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.430834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-data\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.430921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.430867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb07e50-b590-4a0c-8244-c45d83329b75-istio-podinfo\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.430921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.430894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-podinfo\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431141 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.430959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-credential-socket\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431141 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.430985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-envoy\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431141 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmw8r\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-kube-api-access-jmw8r\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431141 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-credential-socket\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431141 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-istio-token\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431141 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istiod-ca-cert\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-socket\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-data\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-certs\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-data\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-token\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-socket\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-certs\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-envoy\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.431966 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-credential-socket\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.431966 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.431673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb07e50-b590-4a0c-8244-c45d83329b75-istiod-ca-cert\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.432109 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.432086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-socket\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.432243 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.432220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-certs\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.438960 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.438937 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-envoy\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.439714 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.439533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb07e50-b590-4a0c-8244-c45d83329b75-istio-podinfo\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.444923 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.444821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-istio-token\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.445026 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.444988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmw8r\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-kube-api-access-jmw8r\") pod \"e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.481590 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.481552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.511706 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.511676 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.533483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-podinfo\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-credential-socket\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istiod-ca-cert\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-data\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-certs\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-token\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-socket\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-envoy\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp85p\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-kube-api-access-sp85p\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.534774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-credential-socket\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.535063 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-certs\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.535615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-socket\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.535811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.535767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istiod-ca-cert\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.536640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.536059 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-data\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.537851 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.537807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-podinfo\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.539228 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.539132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-envoy\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.545448 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.545420 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-token\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.545542 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.545503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp85p\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-kube-api-access-sp85p\") pod \"e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.586229 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.586190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:09.616725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.616679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.666082 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.666048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:09.800071 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.800045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5"] Sep 15 20:54:09.802362 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:09.802331 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc431a76_246a_4136_a3eb_a594eba56ed3.slice/crio-5fbbdfee4bdb00f578d7a1c2911a2a163bd52a5adbf2a344c16ef99173f6a271 WatchSource:0}: Error finding container 5fbbdfee4bdb00f578d7a1c2911a2a163bd52a5adbf2a344c16ef99173f6a271: Status 404 returned error can't find the container with id 5fbbdfee4bdb00f578d7a1c2911a2a163bd52a5adbf2a344c16ef99173f6a271 Sep 15 20:54:09.804828 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.804796 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.804938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.804857 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.804938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.804889 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.828308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.828280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:09.866430 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.866391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk"] Sep 15 20:54:09.867169 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:09.867133 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb07e50_b590_4a0c_8244_c45d83329b75.slice/crio-c59a38028277b89cd50e51c7fa438428501539de2e2615f716b113b666aec9b8 WatchSource:0}: Error finding container c59a38028277b89cd50e51c7fa438428501539de2e2615f716b113b666aec9b8: Status 404 returned error can't find the container with id c59a38028277b89cd50e51c7fa438428501539de2e2615f716b113b666aec9b8 Sep 15 20:54:09.869489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.869459 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.869557 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.869521 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.869557 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.869548 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.948548 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.948523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm"] Sep 15 20:54:09.949701 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:09.949677 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b4b1ec_f9af_4b73_a2de_60b17235156f.slice/crio-066ea002ed0cde204aa3b88fd466097f596303fc20d15226b0f92e2473888d75 WatchSource:0}: Error finding container 066ea002ed0cde204aa3b88fd466097f596303fc20d15226b0f92e2473888d75: Status 404 returned error can't find the container with id 066ea002ed0cde204aa3b88fd466097f596303fc20d15226b0f92e2473888d75 Sep 15 20:54:09.952624 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.952593 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.952723 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.952669 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:09.952723 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:09.952713 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:10.229050 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.228959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" event={"ID":"edb07e50-b590-4a0c-8244-c45d83329b75","Type":"ContainerStarted","Data":"96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604"} Sep 15 20:54:10.229050 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.229016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" event={"ID":"edb07e50-b590-4a0c-8244-c45d83329b75","Type":"ContainerStarted","Data":"c59a38028277b89cd50e51c7fa438428501539de2e2615f716b113b666aec9b8"} Sep 15 20:54:10.231494 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.231330 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" event={"ID":"cc431a76-246a-4136-a3eb-a594eba56ed3","Type":"ContainerStarted","Data":"6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f"} Sep 15 20:54:10.231494 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.231390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" event={"ID":"cc431a76-246a-4136-a3eb-a594eba56ed3","Type":"ContainerStarted","Data":"5fbbdfee4bdb00f578d7a1c2911a2a163bd52a5adbf2a344c16ef99173f6a271"} Sep 15 20:54:10.233466 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.233421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" event={"ID":"a1b4b1ec-f9af-4b73-a2de-60b17235156f","Type":"ContainerStarted","Data":"77e54363eff6eaf37500c22c103b20a49afe83fe9fd505717dd3ef7801dd9e01"} Sep 15 20:54:10.233605 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.233475 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" event={"ID":"a1b4b1ec-f9af-4b73-a2de-60b17235156f","Type":"ContainerStarted","Data":"066ea002ed0cde204aa3b88fd466097f596303fc20d15226b0f92e2473888d75"} Sep 15 20:54:10.246343 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.246278 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podStartSLOduration=1.2462584859999999 podStartE2EDuration="1.246258486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:10.246113737 +0000 UTC m=+1039.375850321" watchObservedRunningTime="2026-09-15 20:54:10.246258486 +0000 UTC m=+1039.375995071" Sep 15 20:54:10.264445 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.264377 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podStartSLOduration=1.264363115 podStartE2EDuration="1.264363115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:10.263287438 +0000 UTC m=+1039.393024022" watchObservedRunningTime="2026-09-15 20:54:10.264363115 +0000 UTC m=+1039.394099697" Sep 15 20:54:10.281241 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.281199 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podStartSLOduration=1.281184863 podStartE2EDuration="1.281184863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:10.278761162 +0000 UTC m=+1039.408497744" watchObservedRunningTime="2026-09-15 20:54:10.281184863 +0000 UTC m=+1039.410921446" Sep 15 20:54:10.455318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.455279 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:10.512386 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.512345 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:10.517586 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.517561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:10.578341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.578309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:10.586863 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.586845 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:10.591689 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.591669 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:10.666895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:10.666866 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:11.021340 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.021307 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:11.175640 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.175610 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7764-a6eb57"] Sep 15 20:54:11.179306 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.179290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:11.181596 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.181575 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cs2t2\"" Sep 15 20:54:11.185948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.185923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-a6eb57"] Sep 15 20:54:11.241377 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.241347 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:11.241592 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.241400 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:11.242224 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.242203 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:11.242340 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.242251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:11.254182 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.254159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wpzm\" (UniqueName: \"kubernetes.io/projected/9a629170-5c26-4cd6-945b-6fb2603c55ec-kube-api-access-4wpzm\") pod \"test-curl-7764-a6eb57\" (UID: \"9a629170-5c26-4cd6-945b-6fb2603c55ec\") " pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:11.355258 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.355162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wpzm\" (UniqueName: \"kubernetes.io/projected/9a629170-5c26-4cd6-945b-6fb2603c55ec-kube-api-access-4wpzm\") pod \"test-curl-7764-a6eb57\" (UID: \"9a629170-5c26-4cd6-945b-6fb2603c55ec\") " pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:11.364931 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.364905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wpzm\" (UniqueName: \"kubernetes.io/projected/9a629170-5c26-4cd6-945b-6fb2603c55ec-kube-api-access-4wpzm\") pod \"test-curl-7764-a6eb57\" (UID: \"9a629170-5c26-4cd6-945b-6fb2603c55ec\") " pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:11.491279 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.491250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:11.672208 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.672134 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:11.766095 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.766063 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-a6eb57"] Sep 15 20:54:11.768166 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:11.768139 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a629170_5c26_4cd6_945b_6fb2603c55ec.slice/crio-47b7c01741d8c532f9fa469a01ed708282c982756cd5e2fc4b8c56c75bc1b8b7 WatchSource:0}: Error finding container 47b7c01741d8c532f9fa469a01ed708282c982756cd5e2fc4b8c56c75bc1b8b7: Status 404 returned error can't find the container with id 47b7c01741d8c532f9fa469a01ed708282c982756cd5e2fc4b8c56c75bc1b8b7 Sep 15 20:54:11.770473 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:11.770454 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:54:12.249422 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:12.249376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-a6eb57" event={"ID":"9a629170-5c26-4cd6-945b-6fb2603c55ec","Type":"ContainerStarted","Data":"47b7c01741d8c532f9fa469a01ed708282c982756cd5e2fc4b8c56c75bc1b8b7"} Sep 15 20:54:12.250061 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:12.250020 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:12.251020 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:12.250997 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:12.278147 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:12.278120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:12.379671 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:12.379635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:12.475925 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:12.475894 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:13.259973 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:13.259940 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-a6eb57" event={"ID":"9a629170-5c26-4cd6-945b-6fb2603c55ec","Type":"ContainerStarted","Data":"43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac"} Sep 15 20:54:13.281837 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:13.281786 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7764-a6eb57" podStartSLOduration=1.553548727 podStartE2EDuration="2.28177054s" totalImagesPullingTime="728.221813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:13.279799776 +0000 UTC m=+1042.409536365" watchObservedRunningTime="2026-09-15 20:54:13.28177054 +0000 UTC m=+1042.411507151" Sep 15 20:54:13.464888 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:13.464858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:13.565385 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:13.565304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:13.663985 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:13.663959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:14.117386 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.117348 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7764-a6eb57"] Sep 15 20:54:14.122242 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.122209 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7764-a6eb57"] Sep 15 20:54:14.145831 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.145798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:14.230892 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.230848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:14.246663 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.246436 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7764-a92172"] Sep 15 20:54:14.247199 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.247164 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" containerName="test-curl-7764-a6eb57" Sep 15 20:54:14.247199 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.247191 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" containerName="test-curl-7764-a6eb57" Sep 15 20:54:14.247450 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.247351 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" containerName="test-curl-7764-a6eb57" Sep 15 20:54:14.251217 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.251178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:14.259384 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.259144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-a92172"] Sep 15 20:54:14.267114 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:14.267092 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7764-a6eb57" container="test-curl-7764-a6eb57" Sep 15 20:54:14.387855 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.387764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmfq\" (UniqueName: \"kubernetes.io/projected/c14b03b3-4388-4ecd-a245-9962a4660930-kube-api-access-rnmfq\") pod \"test-curl-7764-a92172\" (UID: \"c14b03b3-4388-4ecd-a245-9962a4660930\") " pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:14.434603 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.434560 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:14.489100 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.489056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmfq\" (UniqueName: \"kubernetes.io/projected/c14b03b3-4388-4ecd-a245-9962a4660930-kube-api-access-rnmfq\") pod \"test-curl-7764-a92172\" (UID: \"c14b03b3-4388-4ecd-a245-9962a4660930\") " pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:14.496707 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.496669 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmfq\" (UniqueName: \"kubernetes.io/projected/c14b03b3-4388-4ecd-a245-9962a4660930-kube-api-access-rnmfq\") pod \"test-curl-7764-a92172\" (UID: \"c14b03b3-4388-4ecd-a245-9962a4660930\") " pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:14.564221 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.564182 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:14.850766 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:14.850729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-a92172"] Sep 15 20:54:14.858804 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:14.858709 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc14b03b3_4388_4ecd_a245_9962a4660930.slice/crio-33335d2dbf8fac1bfb6d66bc57196233b8b36b193453e8324cc0ca960960dc45 WatchSource:0}: Error finding container 33335d2dbf8fac1bfb6d66bc57196233b8b36b193453e8324cc0ca960960dc45: Status 404 returned error can't find the container with id 33335d2dbf8fac1bfb6d66bc57196233b8b36b193453e8324cc0ca960960dc45 Sep 15 20:54:15.219469 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.219367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:15.274012 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.273981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-a92172" event={"ID":"c14b03b3-4388-4ecd-a245-9962a4660930","Type":"ContainerStarted","Data":"79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84"} Sep 15 20:54:15.274493 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.274025 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-a92172" event={"ID":"c14b03b3-4388-4ecd-a245-9962a4660930","Type":"ContainerStarted","Data":"33335d2dbf8fac1bfb6d66bc57196233b8b36b193453e8324cc0ca960960dc45"} Sep 15 20:54:15.274493 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.274224 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7764-a6eb57" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" containerName="test-curl-7764-a6eb57" containerID="cri-o://43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac" gracePeriod=2 Sep 15 20:54:15.290610 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:15.290562 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a6eb57\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:15.290748 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.290722 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7764-a92172" podStartSLOduration=1.290710455 podStartE2EDuration="1.290710455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:15.288564234 +0000 UTC m=+1044.418300818" watchObservedRunningTime="2026-09-15 20:54:15.290710455 +0000 UTC m=+1044.420447037" Sep 15 20:54:15.292913 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.292602 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:15.348206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.348173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:15.664159 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.664128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:15.750030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.749997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:15.924597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.924498 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7764-a92172"] Sep 15 20:54:15.927649 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:15.927617 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7764-a92172"] Sep 15 20:54:16.036922 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.036884 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p"] Sep 15 20:54:16.037598 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.037573 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" containerName="test-curl-7764-a92172" Sep 15 20:54:16.037598 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.037596 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" containerName="test-curl-7764-a92172" Sep 15 20:54:16.037760 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.037707 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" containerName="test-curl-7764-a92172" Sep 15 20:54:16.041584 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.041564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.043438 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.043398 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-3f1c93-serving-cert\"" Sep 15 20:54:16.043578 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.043478 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 20:54:16.044155 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.044134 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-crdcx\"" Sep 15 20:54:16.044263 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.044227 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 20:54:16.044327 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.044287 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 20:54:16.050439 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.050142 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p"] Sep 15 20:54:16.106757 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.106721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2stw\" (UniqueName: \"kubernetes.io/projected/aa3d2622-0cac-4f1a-86fa-20a6613e4811-kube-api-access-m2stw\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.106928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.106773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3d2622-0cac-4f1a-86fa-20a6613e4811-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.106928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.106799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.106928 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.106845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-metrics-certs\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.109890 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.109866 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:16.208364 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.208272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2stw\" (UniqueName: \"kubernetes.io/projected/aa3d2622-0cac-4f1a-86fa-20a6613e4811-kube-api-access-m2stw\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.208364 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.208321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3d2622-0cac-4f1a-86fa-20a6613e4811-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.208364 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.208348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.208685 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.208382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-metrics-certs\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.208950 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.208927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3d2622-0cac-4f1a-86fa-20a6613e4811-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.211029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.211000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-metrics-certs\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.211163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.211043 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.222161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.222139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2stw\" (UniqueName: \"kubernetes.io/projected/aa3d2622-0cac-4f1a-86fa-20a6613e4811-kube-api-access-m2stw\") pod \"maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.230056 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.230022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:16.280435 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:16.280393 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7764-a92172" container="test-curl-7764-a92172" Sep 15 20:54:16.325579 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.325546 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:16.354601 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.354575 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:16.528908 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.528870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:16.599219 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.599186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:16.635092 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.635069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw"] Sep 15 20:54:16.637341 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:16.637314 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3d2622_0cac_4f1a_86fa_20a6613e4811.slice/crio-3f520b05f84b577b35dd1df002b5e4871fb42ae30f3e3472c9200072807b6e70 WatchSource:0}: Error finding container 3f520b05f84b577b35dd1df002b5e4871fb42ae30f3e3472c9200072807b6e70: Status 404 returned error can't find the container with id 3f520b05f84b577b35dd1df002b5e4871fb42ae30f3e3472c9200072807b6e70 Sep 15 20:54:16.639921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.639904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p"] Sep 15 20:54:16.640010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.639995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.642006 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.641986 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3f1c93-dockercfg-p2rht\"" Sep 15 20:54:16.642084 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.642022 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3f1c93\"" Sep 15 20:54:16.647241 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.647220 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw"] Sep 15 20:54:16.714004 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.713964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a0cf-01c3-4c36-8f7a-485107ebd453-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.714162 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.714023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbsx\" (UniqueName: \"kubernetes.io/projected/0180a0cf-01c3-4c36-8f7a-485107ebd453-kube-api-access-bbbsx\") pod \"payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.815918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.815826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a0cf-01c3-4c36-8f7a-485107ebd453-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.816193 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.816162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbsx\" (UniqueName: \"kubernetes.io/projected/0180a0cf-01c3-4c36-8f7a-485107ebd453-kube-api-access-bbbsx\") pod \"payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.816768 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.816742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a0cf-01c3-4c36-8f7a-485107ebd453-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.824929 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.824903 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f"] Sep 15 20:54:16.828692 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.828669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.830793 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.830775 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b9ff68-openshift-default-dockercfg-kgff6\"" Sep 15 20:54:16.831570 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.831545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbsx\" (UniqueName: \"kubernetes.io/projected/0180a0cf-01c3-4c36-8f7a-485107ebd453-kube-api-access-bbbsx\") pod \"payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.846252 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.846226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f"] Sep 15 20:54:16.860712 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.860674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:16.917376 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-credential-socket\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917534 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-podinfo\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917534 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-socket\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917625 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-data\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917625 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdk7\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-kube-api-access-nsdk7\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917704 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-envoy\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917745 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istiod-ca-cert\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917784 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-certs\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.917818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.917793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-token\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:16.952336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.952302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:16.956612 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:16.956547 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.021732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-data\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.021797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdk7\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-kube-api-access-nsdk7\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.021836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-envoy\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.021870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istiod-ca-cert\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.021926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-certs\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.021963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-token\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.021992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-credential-socket\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.022065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-podinfo\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.022122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-socket\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.022572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-socket\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.023421 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.022824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-data\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.024954 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.024928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istiod-ca-cert\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.028469 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.027715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-credential-socket\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.031131 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.027931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-certs\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.051171 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.051105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-podinfo\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.053463 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.052728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-envoy\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.053463 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.053389 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdk7\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-kube-api-access-nsdk7\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.058890 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.058765 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-token\") pod \"e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.102595 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.102515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:17.139465 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.139395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:17.227215 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.226973 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw"] Sep 15 20:54:17.290111 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.290080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" event={"ID":"aa3d2622-0cac-4f1a-86fa-20a6613e4811","Type":"ContainerStarted","Data":"3f520b05f84b577b35dd1df002b5e4871fb42ae30f3e3472c9200072807b6e70"} Sep 15 20:54:17.291773 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.291737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" event={"ID":"0180a0cf-01c3-4c36-8f7a-485107ebd453","Type":"ContainerStarted","Data":"ac752d75ca5dd115306d452afd9a7f862fa3854fda3782ad4bf7e9033535fc57"} Sep 15 20:54:17.292125 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.292095 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7764-a92172" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" containerName="test-curl-7764-a92172" containerID="cri-o://79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84" gracePeriod=2 Sep 15 20:54:17.293862 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:17.293833 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a92172\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:17.421390 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.421359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f"] Sep 15 20:54:17.432688 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:17.432514 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd53285b_d5c3_4c10_9979_72d35f95b2b7.slice/crio-867b48228c291fc10524993a07683b99ebd7495e8c35449db72c70a1b848f2a1 WatchSource:0}: Error finding container 867b48228c291fc10524993a07683b99ebd7495e8c35449db72c70a1b848f2a1: Status 404 returned error can't find the container with id 867b48228c291fc10524993a07683b99ebd7495e8c35449db72c70a1b848f2a1 Sep 15 20:54:17.440286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.438771 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:17.440286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.438864 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:17.440286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.438901 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:17.489881 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.489858 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:17.630741 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.630156 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wpzm\" (UniqueName: \"kubernetes.io/projected/9a629170-5c26-4cd6-945b-6fb2603c55ec-kube-api-access-4wpzm\") pod \"9a629170-5c26-4cd6-945b-6fb2603c55ec\" (UID: \"9a629170-5c26-4cd6-945b-6fb2603c55ec\") " Sep 15 20:54:17.640038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.639924 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a629170-5c26-4cd6-945b-6fb2603c55ec-kube-api-access-4wpzm" (OuterVolumeSpecName: "kube-api-access-4wpzm") pod "9a629170-5c26-4cd6-945b-6fb2603c55ec" (UID: "9a629170-5c26-4cd6-945b-6fb2603c55ec"). InnerVolumeSpecName "kube-api-access-4wpzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:54:17.732050 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.731965 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wpzm\" (UniqueName: \"kubernetes.io/projected/9a629170-5c26-4cd6-945b-6fb2603c55ec-kube-api-access-4wpzm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:17.956311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:17.956183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:18.075353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.075310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:18.302767 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.302728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" event={"ID":"0180a0cf-01c3-4c36-8f7a-485107ebd453","Type":"ContainerStarted","Data":"a7a4e62f086523c2cc903d17e3c2238f1ac1dd6828f5b0943bcb8885dfc3e692"} Sep 15 20:54:18.303248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.302800 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:18.304560 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.304525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" event={"ID":"cd53285b-d5c3-4c10-9979-72d35f95b2b7","Type":"ContainerStarted","Data":"b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135"} Sep 15 20:54:18.304710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.304590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" event={"ID":"cd53285b-d5c3-4c10-9979-72d35f95b2b7","Type":"ContainerStarted","Data":"867b48228c291fc10524993a07683b99ebd7495e8c35449db72c70a1b848f2a1"} Sep 15 20:54:18.305969 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.305943 2580 generic.go:356] "Generic (PLEG): container finished" podID="9a629170-5c26-4cd6-945b-6fb2603c55ec" containerID="43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac" exitCode=137 Sep 15 20:54:18.306088 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.305971 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:18.306088 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.306038 2580 scope.go:117] "RemoveContainer" containerID="43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac" Sep 15 20:54:18.317249 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.316650 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" podStartSLOduration=2.316609888 podStartE2EDuration="2.316609888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:18.316198358 +0000 UTC m=+1047.445934943" watchObservedRunningTime="2026-09-15 20:54:18.316609888 +0000 UTC m=+1047.446346473" Sep 15 20:54:18.336482 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:18.336437 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a6eb57\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:18.336844 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.336791 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podStartSLOduration=2.336775533 podStartE2EDuration="2.336775533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:18.334551798 +0000 UTC m=+1047.464288393" watchObservedRunningTime="2026-09-15 20:54:18.336775533 +0000 UTC m=+1047.466512120" Sep 15 20:54:18.562206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.562158 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g"] Sep 15 20:54:18.572568 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.572543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.574966 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.574930 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ded67cd1-openshift-default-dockercfg-sp5t7\"" Sep 15 20:54:18.577003 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.576979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g"] Sep 15 20:54:18.597610 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:18.597569 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a6eb57\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:18.642632 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.642598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-credential-socket\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.642798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.642647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-envoy\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.642798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.642766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2c7740-8755-485d-8f31-c1957ca93363-istio-podinfo\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.642903 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.642810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-istio-token\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.642903 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.642866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-data\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.643000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.642945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7l7r\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-kube-api-access-f7l7r\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.643062 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.643013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2c7740-8755-485d-8f31-c1957ca93363-istiod-ca-cert\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.643104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.643068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-certs\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.643104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.643095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-socket\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.658204 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.658172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-credential-socket\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-envoy\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2c7740-8755-485d-8f31-c1957ca93363-istio-podinfo\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-istio-token\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-data\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7l7r\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-kube-api-access-f7l7r\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2c7740-8755-485d-8f31-c1957ca93363-istiod-ca-cert\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.745984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-certs\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.746017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-socket\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.746119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-credential-socket\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.746371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.746330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-socket\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.747268 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.746898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-data\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.747268 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.747147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-certs\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.747582 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.747512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2c7740-8755-485d-8f31-c1957ca93363-istiod-ca-cert\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.750673 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.750585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-envoy\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.752349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.752323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2c7740-8755-485d-8f31-c1957ca93363-istio-podinfo\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.756999 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.756946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-istio-token\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.757310 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.757280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7l7r\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-kube-api-access-f7l7r\") pod \"e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:18.782606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.782560 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:18.887057 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:18.886971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:19.033399 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.033360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:19.140183 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.140082 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:19.146645 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.146584 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:19.169551 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:19.169508 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a6eb57\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" pod="opendatahub/test-curl-7764-a6eb57" Sep 15 20:54:19.320217 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.320182 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a629170-5c26-4cd6-945b-6fb2603c55ec" path="/var/lib/kubelet/pods/9a629170-5c26-4cd6-945b-6fb2603c55ec/volumes" Sep 15 20:54:19.320639 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.320622 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:19.320723 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.320705 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:19.388176 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.387394 2580 scope.go:117] "RemoveContainer" containerID="43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac" Sep 15 20:54:19.388333 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:19.388274 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac\": container with ID starting with 43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac not found: ID does not exist" containerID="43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac" Sep 15 20:54:19.388333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.388316 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac"} err="failed to get container status \"43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac\": rpc error: code = NotFound desc = could not find container \"43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac\": container with ID starting with 43fa6dc80d0134eac8819f8746f4583b7dac30021fb30e826dec46b3d29162ac not found: ID does not exist" Sep 15 20:54:19.548584 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.548528 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:19.614774 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.614737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:19.658357 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.658274 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnmfq\" (UniqueName: \"kubernetes.io/projected/c14b03b3-4388-4ecd-a245-9962a4660930-kube-api-access-rnmfq\") pod \"c14b03b3-4388-4ecd-a245-9962a4660930\" (UID: \"c14b03b3-4388-4ecd-a245-9962a4660930\") " Sep 15 20:54:19.660519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.660495 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14b03b3-4388-4ecd-a245-9962a4660930-kube-api-access-rnmfq" (OuterVolumeSpecName: "kube-api-access-rnmfq") pod "c14b03b3-4388-4ecd-a245-9962a4660930" (UID: "c14b03b3-4388-4ecd-a245-9962a4660930"). InnerVolumeSpecName "kube-api-access-rnmfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:54:19.686881 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.686845 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g"] Sep 15 20:54:19.688606 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:19.688582 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2c7740_8755_485d_8f31_c1957ca93363.slice/crio-a4079046aa8b8c740c3390bb6c3f7c41975adf93abf10b8b6d0ff28336628548 WatchSource:0}: Error finding container a4079046aa8b8c740c3390bb6c3f7c41975adf93abf10b8b6d0ff28336628548: Status 404 returned error can't find the container with id a4079046aa8b8c740c3390bb6c3f7c41975adf93abf10b8b6d0ff28336628548 Sep 15 20:54:19.691118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.691088 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:19.691182 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.691151 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:19.691182 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.691179 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:19.759973 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:19.759950 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmfq\" (UniqueName: \"kubernetes.io/projected/c14b03b3-4388-4ecd-a245-9962a4660930-kube-api-access-rnmfq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:20.303033 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.302997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:20.324614 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.324584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" event={"ID":"cd2c7740-8755-485d-8f31-c1957ca93363","Type":"ContainerStarted","Data":"e8a68c8bc489413b7bf1dfd53e65ac42c486660d78ad35c0219faa84502422a2"} Sep 15 20:54:20.325085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.324636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" event={"ID":"cd2c7740-8755-485d-8f31-c1957ca93363","Type":"ContainerStarted","Data":"a4079046aa8b8c740c3390bb6c3f7c41975adf93abf10b8b6d0ff28336628548"} Sep 15 20:54:20.326503 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.326474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" event={"ID":"aa3d2622-0cac-4f1a-86fa-20a6613e4811","Type":"ContainerStarted","Data":"79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc"} Sep 15 20:54:20.326642 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.326611 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:20.329362 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.329337 2580 generic.go:356] "Generic (PLEG): container finished" podID="c14b03b3-4388-4ecd-a245-9962a4660930" containerID="79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84" exitCode=137 Sep 15 20:54:20.329481 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.329389 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:20.329481 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.329448 2580 scope.go:117] "RemoveContainer" containerID="79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84" Sep 15 20:54:20.342488 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.342471 2580 scope.go:117] "RemoveContainer" containerID="79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84" Sep 15 20:54:20.342776 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:20.342757 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84\": container with ID starting with 79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84 not found: ID does not exist" containerID="79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84" Sep 15 20:54:20.342886 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.342786 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84"} err="failed to get container status \"79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84\": rpc error: code = NotFound desc = could not find container \"79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84\": container with ID starting with 79f05908483f9bbfaba35945661d1492d4c9adfe4aad3d64f2957fdec358cb84 not found: ID does not exist" Sep 15 20:54:20.351552 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:20.351525 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a92172\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:20.351884 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.351849 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" podStartSLOduration=2.351836321 podStartE2EDuration="2.351836321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:20.349866186 +0000 UTC m=+1049.479602808" watchObservedRunningTime="2026-09-15 20:54:20.351836321 +0000 UTC m=+1049.481572903" Sep 15 20:54:20.371307 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.371264 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" podStartSLOduration=1.546574833 podStartE2EDuration="4.371250471s" totalImagesPullingTime="2.824675638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:20.368878028 +0000 UTC m=+1049.498614612" watchObservedRunningTime="2026-09-15 20:54:20.371250471 +0000 UTC m=+1049.500987078" Sep 15 20:54:20.373317 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.373283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:20.731818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.731702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:20.887516 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.887481 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:20.892587 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:20.892556 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:20.915387 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:20.915345 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a92172\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:21.312939 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:21.312897 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-a92172\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" pod="opendatahub/test-curl-7764-a92172" Sep 15 20:54:21.316096 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:21.316065 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14b03b3-4388-4ecd-a245-9962a4660930" path="/var/lib/kubelet/pods/c14b03b3-4388-4ecd-a245-9962a4660930/volumes" Sep 15 20:54:21.337859 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:21.337832 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:21.338674 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:21.338653 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:21.649502 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:21.649418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:21.745677 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:21.745645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:21.844620 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:21.844586 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:21.991124 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:21.991044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:22.352855 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:22.352816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:22.533051 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:22.533005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:22.686187 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:22.686101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:23.165800 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.165763 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:23.330503 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.330471 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t"] Sep 15 20:54:23.336743 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.336717 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t"] Sep 15 20:54:23.337005 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.336987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.340854 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.340117 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-b9ff68-serving-cert\"" Sep 15 20:54:23.363836 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.363798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:23.467805 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.467673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:23.501666 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.501006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a51588-f8bd-42e2-93c9-d88220bd7f59-openshift-service-ca\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.501666 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.501100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.501666 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.501167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-metrics-certs\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.501666 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.501195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhbs\" (UniqueName: \"kubernetes.io/projected/91a51588-f8bd-42e2-93c9-d88220bd7f59-kube-api-access-hkhbs\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.602066 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.602018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.602265 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.602094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-metrics-certs\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.602265 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.602128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhbs\" (UniqueName: \"kubernetes.io/projected/91a51588-f8bd-42e2-93c9-d88220bd7f59-kube-api-access-hkhbs\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.602265 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:23.602192 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-serving-cert: secret "maas-api-e2e-autores-b9ff68-serving-cert" not found Sep 15 20:54:23.602502 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:23.602279 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls podName:91a51588-f8bd-42e2-93c9-d88220bd7f59 nodeName:}" failed. No retries permitted until 2026-09-15 20:54:24.102254707 +0000 UTC m=+1053.231991269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls") pod "maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" (UID: "91a51588-f8bd-42e2-93c9-d88220bd7f59") : secret "maas-api-e2e-autores-b9ff68-serving-cert" not found Sep 15 20:54:23.602502 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.602305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a51588-f8bd-42e2-93c9-d88220bd7f59-openshift-service-ca\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.603142 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.603115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a51588-f8bd-42e2-93c9-d88220bd7f59-openshift-service-ca\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.606023 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.605992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-metrics-certs\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.614344 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.614262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhbs\" (UniqueName: \"kubernetes.io/projected/91a51588-f8bd-42e2-93c9-d88220bd7f59-kube-api-access-hkhbs\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:23.827438 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.826382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:23.926500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:23.926321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:24.009426 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:24.008218 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:24.111504 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:24.111296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:24.137432 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:24.136520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls\") pod \"maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:24.267430 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:24.266629 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:24.433399 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:24.433349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:24.708284 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:24.708200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t"] Sep 15 20:54:25.142056 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.142009 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp"] Sep 15 20:54:25.149718 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.149688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.152370 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.152176 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ded67cd1-serving-cert\"" Sep 15 20:54:25.169978 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.169790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:25.182879 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.179086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp"] Sep 15 20:54:25.234240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.233843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-maas-api-tls\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.234240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.233916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-metrics-certs\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.234240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.233997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-openshift-service-ca\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.234240 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.234141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887lw\" (UniqueName: \"kubernetes.io/projected/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-kube-api-access-887lw\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.284216 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.284176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:25.353792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.353017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-openshift-service-ca\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.353792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.353176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887lw\" (UniqueName: \"kubernetes.io/projected/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-kube-api-access-887lw\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.353792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.353212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-maas-api-tls\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.353792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.353261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-metrics-certs\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.355392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.355365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-openshift-service-ca\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.366422 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.366326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-metrics-certs\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.368482 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.367027 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-887lw\" (UniqueName: \"kubernetes.io/projected/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-kube-api-access-887lw\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.368482 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.367771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-maas-api-tls\") pod \"maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.372507 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.370923 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:25.391246 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.391171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" event={"ID":"91a51588-f8bd-42e2-93c9-d88220bd7f59","Type":"ContainerStarted","Data":"7bd4353b57dc52ae0a5d36cf1514080a2a8047a54f836ed79593b0c24a8be7f1"} Sep 15 20:54:25.485747 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.485116 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:25.614651 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.614051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:25.728092 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.718261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:25.968429 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:25.967973 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp"] Sep 15 20:54:25.989778 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:25.985710 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda808920b_fabe_40d2_b2e0_f3ae491b4f4c.slice/crio-a519dede2f935354d18efe112208169df46e97da49d16e25ea8d89a30686a799 WatchSource:0}: Error finding container a519dede2f935354d18efe112208169df46e97da49d16e25ea8d89a30686a799: Status 404 returned error can't find the container with id a519dede2f935354d18efe112208169df46e97da49d16e25ea8d89a30686a799 Sep 15 20:54:26.140284 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:26.140244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:26.363430 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:26.362609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 20:54:26.406366 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:26.406323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" event={"ID":"91a51588-f8bd-42e2-93c9-d88220bd7f59","Type":"ContainerStarted","Data":"d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b"} Sep 15 20:54:26.406592 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:26.406578 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:26.408431 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:26.408373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" event={"ID":"a808920b-fabe-40d2-b2e0-f3ae491b4f4c","Type":"ContainerStarted","Data":"a519dede2f935354d18efe112208169df46e97da49d16e25ea8d89a30686a799"} Sep 15 20:54:26.431382 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:26.431316 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" podStartSLOduration=2.8624401969999997 podStartE2EDuration="3.431291193s" totalImagesPullingTime="568.850996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:26.427810815 +0000 UTC m=+1055.557547400" watchObservedRunningTime="2026-09-15 20:54:26.431291193 +0000 UTC m=+1055.561027776" Sep 15 20:54:26.931328 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:26.931156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:27.421768 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:27.421532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" event={"ID":"a808920b-fabe-40d2-b2e0-f3ae491b4f4c","Type":"ContainerStarted","Data":"ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520"} Sep 15 20:54:27.421768 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:27.421700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:27.444435 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:27.444355 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" podStartSLOduration=1.994643606 podStartE2EDuration="2.444334121s" totalImagesPullingTime="449.690515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:27.440358458 +0000 UTC m=+1056.570095057" watchObservedRunningTime="2026-09-15 20:54:27.444334121 +0000 UTC m=+1056.574070707" Sep 15 20:54:27.722178 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:27.722081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:28.288337 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:28.288289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:28.388216 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:28.388172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:28.739225 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:28.739105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:28.833791 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:28.833757 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:29.317480 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:29.317447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 20:54:29.330123 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:29.329810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:30.297438 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.293895 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8"] Sep 15 20:54:30.300642 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.300013 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.303525 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.303091 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-988a09-serving-cert\"" Sep 15 20:54:30.308369 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.308345 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8"] Sep 15 20:54:30.328080 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.328005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:30.379399 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.379354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-metrics-certs\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.379399 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.379415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.379671 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.379499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f05f58-7565-43f5-916e-1b4476f14add-openshift-service-ca\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.379671 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.379564 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lb2q\" (UniqueName: \"kubernetes.io/projected/00f05f58-7565-43f5-916e-1b4476f14add-kube-api-access-4lb2q\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.429066 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.428961 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:30.492125 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.492081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f05f58-7565-43f5-916e-1b4476f14add-openshift-service-ca\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.492305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.492192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lb2q\" (UniqueName: \"kubernetes.io/projected/00f05f58-7565-43f5-916e-1b4476f14add-kube-api-access-4lb2q\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.492366 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.492328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-metrics-certs\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.492366 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.492356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.492573 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:30.492550 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-serving-cert: secret "maas-api-e2e-inf-b-988a09-serving-cert" not found Sep 15 20:54:30.493150 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:30.493134 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls podName:00f05f58-7565-43f5-916e-1b4476f14add nodeName:}" failed. No retries permitted until 2026-09-15 20:54:30.993096263 +0000 UTC m=+1060.122832847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls") pod "maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" (UID: "00f05f58-7565-43f5-916e-1b4476f14add") : secret "maas-api-e2e-inf-b-988a09-serving-cert" not found Sep 15 20:54:30.493371 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.492914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f05f58-7565-43f5-916e-1b4476f14add-openshift-service-ca\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.496280 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.496231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-metrics-certs\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.505238 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.505176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lb2q\" (UniqueName: \"kubernetes.io/projected/00f05f58-7565-43f5-916e-1b4476f14add-kube-api-access-4lb2q\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:30.530847 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.530816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:30.649621 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.649477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:30.732901 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.732834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:30.832118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.832086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:30.933605 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:30.933498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:31.000440 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:31.000156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:31.004299 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:31.004258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls\") pod \"maas-api-e2e-inf-b-988a09-8648c878c7-d99f8\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:31.024199 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:31.024143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:31.226441 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:31.226316 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:31.330072 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:31.324756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:31.538757 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:31.538569 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8"] Sep 15 20:54:31.540394 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:31.540352 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f05f58_7565_43f5_916e_1b4476f14add.slice/crio-eae0010720f20b0118950adac9f187d4fe021502a013f367411d233e5d95cbf6 WatchSource:0}: Error finding container eae0010720f20b0118950adac9f187d4fe021502a013f367411d233e5d95cbf6: Status 404 returned error can't find the container with id eae0010720f20b0118950adac9f187d4fe021502a013f367411d233e5d95cbf6 Sep 15 20:54:32.186931 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.186893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:32.287610 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.287548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:32.427544 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.427512 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:54:32.485490 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.485382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" event={"ID":"00f05f58-7565-43f5-916e-1b4476f14add","Type":"ContainerStarted","Data":"85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492"} Sep 15 20:54:32.485778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.485749 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" event={"ID":"00f05f58-7565-43f5-916e-1b4476f14add","Type":"ContainerStarted","Data":"eae0010720f20b0118950adac9f187d4fe021502a013f367411d233e5d95cbf6"} Sep 15 20:54:32.486192 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.486162 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:32.505436 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.505359 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" podStartSLOduration=1.787270911 podStartE2EDuration="2.5053372s" totalImagesPullingTime="718.066289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:32.500710726 +0000 UTC m=+1061.630447309" watchObservedRunningTime="2026-09-15 20:54:32.5053372 +0000 UTC m=+1061.635073785" Sep 15 20:54:32.662694 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.662654 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:32.765470 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.765436 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:32.869788 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.865127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:34.936932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:32.962218 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:34.936932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:33.559981 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:34.936932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:33.806554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:34.936932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:34.443687 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:34.936932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:34.623966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:34.988273 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:34.988242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:35.447719 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.447685 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:35.508285 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.508256 2580 generic.go:356] "Generic (PLEG): container finished" podID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerID="6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f" exitCode=0 Sep 15 20:54:35.508483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.508325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" event={"ID":"cc431a76-246a-4136-a3eb-a594eba56ed3","Type":"ContainerDied","Data":"6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f"} Sep 15 20:54:35.508861 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.508835 2580 scope.go:117] "RemoveContainer" containerID="6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f" Sep 15 20:54:35.510045 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.510010 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.510147 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.510097 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.510194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.510144 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.510275 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.510257 2580 generic.go:356] "Generic (PLEG): container finished" podID="edb07e50-b590-4a0c-8244-c45d83329b75" containerID="96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604" exitCode=0 Sep 15 20:54:35.510342 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.510322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" event={"ID":"edb07e50-b590-4a0c-8244-c45d83329b75","Type":"ContainerDied","Data":"96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604"} Sep 15 20:54:35.510855 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.510726 2580 scope.go:117] "RemoveContainer" containerID="96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604" Sep 15 20:54:35.512506 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.512462 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.512628 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.512556 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.512628 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.512609 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.514281 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.514108 2580 generic.go:356] "Generic (PLEG): container finished" podID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerID="77e54363eff6eaf37500c22c103b20a49afe83fe9fd505717dd3ef7801dd9e01" exitCode=0 Sep 15 20:54:35.514281 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.514162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" event={"ID":"a1b4b1ec-f9af-4b73-a2de-60b17235156f","Type":"ContainerDied","Data":"77e54363eff6eaf37500c22c103b20a49afe83fe9fd505717dd3ef7801dd9e01"} Sep 15 20:54:35.514565 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.514549 2580 scope.go:117] "RemoveContainer" containerID="77e54363eff6eaf37500c22c103b20a49afe83fe9fd505717dd3ef7801dd9e01" Sep 15 20:54:35.519734 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.519707 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.519833 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.519764 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.519833 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.519794 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:35.614353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:35.614319 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:36.393561 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.393525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:36.531941 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.531899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" event={"ID":"cc431a76-246a-4136-a3eb-a594eba56ed3","Type":"ContainerStarted","Data":"d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74"} Sep 15 20:54:36.536065 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.536027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" event={"ID":"edb07e50-b590-4a0c-8244-c45d83329b75","Type":"ContainerStarted","Data":"5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82"} Sep 15 20:54:36.543873 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.543836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" event={"ID":"a1b4b1ec-f9af-4b73-a2de-60b17235156f","Type":"ContainerStarted","Data":"45cfe7e31fa783b8ac88a920e2a1323d38c7aa7b395b6a45098bbfbcdea6c418"} Sep 15 20:54:36.586761 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.586719 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:36.587969 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.587934 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:36.588111 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.588006 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:36.669208 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.669095 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:36.669208 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.669138 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 15 20:54:36.669208 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:36.669187 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 15 20:54:37.029848 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.029797 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:37.389487 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.389062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:37.512591 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.512556 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:37.514107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.513928 2580 patch_prober.go:28] interesting pod/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:54:37.514107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.513986 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 15 20:54:37.587226 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.587183 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:37.587440 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.587247 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:37.666920 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.666816 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 15 20:54:37.667178 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.667145 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 15 20:54:37.811328 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.811288 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:37.922492 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:37.920785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:38.009431 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.008881 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:38.512344 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.512298 2580 patch_prober.go:28] interesting pod/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:54:38.512571 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.512368 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 15 20:54:38.587146 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.587094 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:38.587701 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.587169 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:38.658900 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.658846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:38.667760 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.667728 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 15 20:54:38.668052 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:38.668004 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 15 20:54:39.476840 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.476781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:39.512506 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.512027 2580 patch_prober.go:28] interesting pod/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:54:39.512506 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.512094 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 15 20:54:39.515039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.514040 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 20:54:39.612971 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.612933 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:39.613599 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.613558 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:39.668991 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.668940 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 15 20:54:39.669159 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:39.669032 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 15 20:54:40.055683 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.055609 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:40.513163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.513110 2580 patch_prober.go:28] interesting pod/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:54:40.513384 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.513190 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 15 20:54:40.587025 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.586972 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:40.587235 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.587050 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:40.666729 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.666691 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 15 20:54:40.667314 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.667279 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 15 20:54:40.814757 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:40.814646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:41.241955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.241856 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:41.242997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.242976 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:41.512877 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.512826 2580 patch_prober.go:28] interesting pod/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:54:41.513067 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.512906 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 15 20:54:41.531452 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.531393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:41.587259 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.587211 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:41.587458 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.587372 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:41.668710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.666804 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 15 20:54:41.668710 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:41.666887 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 15 20:54:42.250023 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.249980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:42.318077 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.318013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:42.512848 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.512804 2580 patch_prober.go:28] interesting pod/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 15 20:54:42.513073 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.512896 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 15 20:54:42.587397 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.587344 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:42.587636 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.587455 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:42.667114 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.667076 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 15 20:54:42.667816 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:42.667390 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 15 20:54:43.085842 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.085789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:43.474612 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.474510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:43.587691 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.587656 2580 patch_prober.go:28] interesting pod/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 15 20:54:43.588005 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.587974 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 15 20:54:43.769878 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.769823 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:43.786706 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.786663 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 20:54:43.818717 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.818677 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:43.825106 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.825076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 20:54:43.900494 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:43.899788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:44.544866 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:44.544830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:44.594733 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:44.594709 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:44.596137 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:44.596113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 20:54:45.266838 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:45.266791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:46.018222 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:46.018187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:46.641923 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:46.641885 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:47.269244 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:47.269194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:47.854468 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:47.854423 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp"] Sep 15 20:54:47.854992 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:47.854952 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" podUID="a808920b-fabe-40d2-b2e0-f3ae491b4f4c" containerName="maas-api" containerID="cri-o://ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520" gracePeriod=30 Sep 15 20:54:47.866914 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:47.866886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:47.927476 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:47.927263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:47.987996 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:47.987958 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:48.052061 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.052030 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:48.107652 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.107528 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:48.187052 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.187009 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g"] Sep 15 20:54:48.187522 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.187486 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" podUID="cd2c7740-8755-485d-8f31-c1957ca93363" containerName="istio-proxy" containerID="cri-o://e8a68c8bc489413b7bf1dfd53e65ac42c486660d78ad35c0219faa84502422a2" gracePeriod=30 Sep 15 20:54:48.202546 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.202074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:48.233674 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.233479 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:48.354850 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.354794 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-887lw\" (UniqueName: \"kubernetes.io/projected/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-kube-api-access-887lw\") pod \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " Sep 15 20:54:48.355333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.354868 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-metrics-certs\") pod \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " Sep 15 20:54:48.355333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.354931 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-openshift-service-ca\") pod \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " Sep 15 20:54:48.355333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.355061 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-maas-api-tls\") pod \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\" (UID: \"a808920b-fabe-40d2-b2e0-f3ae491b4f4c\") " Sep 15 20:54:48.355566 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.355346 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a808920b-fabe-40d2-b2e0-f3ae491b4f4c" (UID: "a808920b-fabe-40d2-b2e0-f3ae491b4f4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:54:48.355677 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.355559 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:48.358329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.358255 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a808920b-fabe-40d2-b2e0-f3ae491b4f4c" (UID: "a808920b-fabe-40d2-b2e0-f3ae491b4f4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:54:48.358329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.358308 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a808920b-fabe-40d2-b2e0-f3ae491b4f4c" (UID: "a808920b-fabe-40d2-b2e0-f3ae491b4f4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:54:48.358533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.358332 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-kube-api-access-887lw" (OuterVolumeSpecName: "kube-api-access-887lw") pod "a808920b-fabe-40d2-b2e0-f3ae491b4f4c" (UID: "a808920b-fabe-40d2-b2e0-f3ae491b4f4c"). InnerVolumeSpecName "kube-api-access-887lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:54:48.457179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.457143 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:48.457179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.457178 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:48.457179 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.457187 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887lw\" (UniqueName: \"kubernetes.io/projected/a808920b-fabe-40d2-b2e0-f3ae491b4f4c-kube-api-access-887lw\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:48.695129 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.695031 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:48.708318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.708273 2580 generic.go:356] "Generic (PLEG): container finished" podID="a808920b-fabe-40d2-b2e0-f3ae491b4f4c" containerID="ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520" exitCode=0 Sep 15 20:54:48.708511 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.708336 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" Sep 15 20:54:48.708511 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.708360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" event={"ID":"a808920b-fabe-40d2-b2e0-f3ae491b4f4c","Type":"ContainerDied","Data":"ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520"} Sep 15 20:54:48.708511 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.708431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp" event={"ID":"a808920b-fabe-40d2-b2e0-f3ae491b4f4c","Type":"ContainerDied","Data":"a519dede2f935354d18efe112208169df46e97da49d16e25ea8d89a30686a799"} Sep 15 20:54:48.708511 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.708463 2580 scope.go:117] "RemoveContainer" containerID="ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520" Sep 15 20:54:48.722945 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.722914 2580 scope.go:117] "RemoveContainer" containerID="ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520" Sep 15 20:54:48.723366 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:48.723333 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520\": container with ID starting with ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520 not found: ID does not exist" containerID="ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520" Sep 15 20:54:48.723451 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.723384 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520"} err="failed to get container status \"ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520\": rpc error: code = NotFound desc = could not find container \"ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520\": container with ID starting with ff6878fdc91b8990805832d3183152431f3641f2e7c2628d1a27cf400668e520 not found: ID does not exist" Sep 15 20:54:48.746728 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.746671 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp"] Sep 15 20:54:48.751718 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.751684 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ded67cd1-799f46c9b-2cptp"] Sep 15 20:54:48.795726 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.795682 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:48.936766 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:48.936724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:49.263714 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.263656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:49.317111 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.317076 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:49.317282 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.317157 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:49.319251 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.319210 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a808920b-fabe-40d2-b2e0-f3ae491b4f4c" path="/var/lib/kubelet/pods/a808920b-fabe-40d2-b2e0-f3ae491b4f4c/volumes" Sep 15 20:54:49.330792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.330760 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:49.564835 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.564736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:49.718245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.718210 2580 generic.go:356] "Generic (PLEG): container finished" podID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerID="b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135" exitCode=0 Sep 15 20:54:49.718772 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.718266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" event={"ID":"cd53285b-d5c3-4c10-9979-72d35f95b2b7","Type":"ContainerDied","Data":"b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135"} Sep 15 20:54:49.718874 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.718773 2580 scope.go:117] "RemoveContainer" containerID="b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135" Sep 15 20:54:49.720129 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.720092 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:49.720250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.720189 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:49.720250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.720238 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:54:49.722115 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.722086 2580 generic.go:356] "Generic (PLEG): container finished" podID="cd2c7740-8755-485d-8f31-c1957ca93363" containerID="e8a68c8bc489413b7bf1dfd53e65ac42c486660d78ad35c0219faa84502422a2" exitCode=0 Sep 15 20:54:49.722216 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.722134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" event={"ID":"cd2c7740-8755-485d-8f31-c1957ca93363","Type":"ContainerDied","Data":"e8a68c8bc489413b7bf1dfd53e65ac42c486660d78ad35c0219faa84502422a2"} Sep 15 20:54:49.866961 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.866823 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:49.974284 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974242 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-istio-token\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.974500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974306 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-socket\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.974500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974335 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-credential-socket\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.974500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974359 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2c7740-8755-485d-8f31-c1957ca93363-istio-podinfo\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.974500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974438 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-certs\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974481 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-data\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974654 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974759 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974884 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-envoy\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974920 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.974940 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7l7r\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-kube-api-access-f7l7r\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.975209 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.975039 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2c7740-8755-485d-8f31-c1957ca93363-istiod-ca-cert\") pod \"cd2c7740-8755-485d-8f31-c1957ca93363\" (UID: \"cd2c7740-8755-485d-8f31-c1957ca93363\") " Sep 15 20:54:49.975699 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.975484 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2c7740-8755-485d-8f31-c1957ca93363-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:54:49.975699 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.975512 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:49.975699 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.975531 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:49.975699 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.975544 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:49.975699 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.975557 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:49.977030 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.977008 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:54:49.977214 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.977192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd2c7740-8755-485d-8f31-c1957ca93363-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 20:54:49.977357 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.977327 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-kube-api-access-f7l7r" (OuterVolumeSpecName: "kube-api-access-f7l7r") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "kube-api-access-f7l7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:54:49.977527 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:49.977512 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd2c7740-8755-485d-8f31-c1957ca93363" (UID: "cd2c7740-8755-485d-8f31-c1957ca93363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:54:50.009158 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.009112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:50.076329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.076289 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2c7740-8755-485d-8f31-c1957ca93363-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:50.076329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.076322 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2c7740-8755-485d-8f31-c1957ca93363-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:50.076329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.076331 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7l7r\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-kube-api-access-f7l7r\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:50.076601 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.076343 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2c7740-8755-485d-8f31-c1957ca93363-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:50.076601 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.076352 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2c7740-8755-485d-8f31-c1957ca93363-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:54:50.630117 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.630016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:50.732618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.732585 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" Sep 15 20:54:50.732618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.732611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g" event={"ID":"cd2c7740-8755-485d-8f31-c1957ca93363","Type":"ContainerDied","Data":"a4079046aa8b8c740c3390bb6c3f7c41975adf93abf10b8b6d0ff28336628548"} Sep 15 20:54:50.732881 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.732688 2580 scope.go:117] "RemoveContainer" containerID="e8a68c8bc489413b7bf1dfd53e65ac42c486660d78ad35c0219faa84502422a2" Sep 15 20:54:50.736138 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.735298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" event={"ID":"cd53285b-d5c3-4c10-9979-72d35f95b2b7","Type":"ContainerStarted","Data":"438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613"} Sep 15 20:54:50.768249 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.768204 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g"] Sep 15 20:54:50.774143 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:50.774107 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ded67cd1-openshift-default-68cb496b7b-n7b7g"] Sep 15 20:54:51.140059 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:51.140015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:51.140311 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:51.140285 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:51.140441 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:51.140360 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:51.316955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:51.316909 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd2c7740-8755-485d-8f31-c1957ca93363" path="/var/lib/kubelet/pods/cd2c7740-8755-485d-8f31-c1957ca93363/volumes" Sep 15 20:54:51.429067 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:51.428958 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:52.093647 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.093611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:52.140085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.140041 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:52.140304 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.140123 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:52.323938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.323892 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:52.425618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.425509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:52.801097 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.801057 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw"] Sep 15 20:54:52.801627 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.801611 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a808920b-fabe-40d2-b2e0-f3ae491b4f4c" containerName="maas-api" Sep 15 20:54:52.801696 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.801629 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a808920b-fabe-40d2-b2e0-f3ae491b4f4c" containerName="maas-api" Sep 15 20:54:52.801696 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.801658 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd2c7740-8755-485d-8f31-c1957ca93363" containerName="istio-proxy" Sep 15 20:54:52.801696 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.801665 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd2c7740-8755-485d-8f31-c1957ca93363" containerName="istio-proxy" Sep 15 20:54:52.801798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.801728 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a808920b-fabe-40d2-b2e0-f3ae491b4f4c" containerName="maas-api" Sep 15 20:54:52.801798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.801738 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd2c7740-8755-485d-8f31-c1957ca93363" containerName="istio-proxy" Sep 15 20:54:52.805232 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.805200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:52.807155 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.807131 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e8af3fa9-serving-cert\"" Sep 15 20:54:52.852346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.852305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw"] Sep 15 20:54:52.907038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.906998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjts\" (UniqueName: \"kubernetes.io/projected/c07aacca-258f-4add-8dff-f66fa54231bb-kube-api-access-czjts\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:52.907038 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.907044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c07aacca-258f-4add-8dff-f66fa54231bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:52.907283 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.907117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:52.907283 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.907195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-metrics-certs\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:52.960141 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:52.960097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:53.008353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.008307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjts\" (UniqueName: \"kubernetes.io/projected/c07aacca-258f-4add-8dff-f66fa54231bb-kube-api-access-czjts\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.008353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.008357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c07aacca-258f-4add-8dff-f66fa54231bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.008600 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.008396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.008600 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.008468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-metrics-certs\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.008600 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:53.008547 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-serving-cert: secret "maas-api-e2e-migrate-e8af3fa9-serving-cert" not found Sep 15 20:54:53.008726 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:54:53.008626 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls podName:c07aacca-258f-4add-8dff-f66fa54231bb nodeName:}" failed. No retries permitted until 2026-09-15 20:54:53.50860653 +0000 UTC m=+1082.638343116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls") pod "maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" (UID: "c07aacca-258f-4add-8dff-f66fa54231bb") : secret "maas-api-e2e-migrate-e8af3fa9-serving-cert" not found Sep 15 20:54:53.009122 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.009102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c07aacca-258f-4add-8dff-f66fa54231bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.022461 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.019194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjts\" (UniqueName: \"kubernetes.io/projected/c07aacca-258f-4add-8dff-f66fa54231bb-kube-api-access-czjts\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.022461 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.019702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-metrics-certs\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.061532 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.061399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:53.140674 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.140623 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:53.141228 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.140700 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:53.175782 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.175737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:53.277772 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.277726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:53.382002 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.381903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:53.526430 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.524397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.540840 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.528567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls\") pod \"maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:53.659880 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.659794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:53.735173 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:53.735132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:54.023093 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.023039 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw"] Sep 15 20:54:54.027924 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:54:54.027880 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07aacca_258f_4add_8dff_f66fa54231bb.slice/crio-a98d25912d2df3de6a01200628595204b9aed7f0a5854fd01fce06f578f829f8 WatchSource:0}: Error finding container a98d25912d2df3de6a01200628595204b9aed7f0a5854fd01fce06f578f829f8: Status 404 returned error can't find the container with id a98d25912d2df3de6a01200628595204b9aed7f0a5854fd01fce06f578f829f8 Sep 15 20:54:54.140763 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.140721 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:54.141134 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.140790 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:54.529889 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.529841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:54.767788 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.767744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" event={"ID":"c07aacca-258f-4add-8dff-f66fa54231bb","Type":"ContainerStarted","Data":"9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6"} Sep 15 20:54:54.768000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.767834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" event={"ID":"c07aacca-258f-4add-8dff-f66fa54231bb","Type":"ContainerStarted","Data":"a98d25912d2df3de6a01200628595204b9aed7f0a5854fd01fce06f578f829f8"} Sep 15 20:54:54.768000 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.767864 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:54:54.783131 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:54.783072 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" podStartSLOduration=2.429285769 podStartE2EDuration="2.783056415s" totalImagesPullingTime="353.770646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:54.781647552 +0000 UTC m=+1083.911384135" watchObservedRunningTime="2026-09-15 20:54:54.783056415 +0000 UTC m=+1083.912793061" Sep 15 20:54:55.140049 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:55.139947 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:55.140349 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:55.140297 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:55.154728 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:55.154692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:55.753723 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:55.753690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:56.140845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:56.140749 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:56.140845 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:56.140813 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:56.440649 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:56.440543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:57.140579 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:57.140528 2580 patch_prober.go:28] interesting pod/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 15 20:54:57.140774 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:57.140628 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 15 20:54:57.969668 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:57.969631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:58.091074 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:58.091028 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:54:58.144824 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:58.144791 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:58.145126 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:58.145106 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:58.145915 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:58.145875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:54:58.241081 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:54:58.240987 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:00.780554 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:00.780524 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:55:04.028146 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:04.028104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:04.136249 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:04.136213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:04.234799 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:04.234767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:04.412151 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:04.412060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:05.454811 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:05.454778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:10.135715 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:10.135675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:10.235964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:10.235925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:14.768271 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:14.768232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:15.418501 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:15.418460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:16.151056 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:16.151019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:16.825596 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:16.825563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:17.648958 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:17.648921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:18.301998 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:18.301965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:18.936261 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:18.936224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:19.731050 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:19.731012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:20.377272 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:20.377221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:21.118086 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:21.118047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:21.757553 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:21.757518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:23.464638 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:23.464601 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:24.005597 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.005559 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw"] Sep 15 20:55:24.005838 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.005806 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" podUID="c07aacca-258f-4add-8dff-f66fa54231bb" containerName="maas-api" containerID="cri-o://9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6" gracePeriod=30 Sep 15 20:55:24.036015 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.035980 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:24.175183 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.175145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:24.240274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.240237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:24.278575 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.278545 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:55:24.366502 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.366389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:24.369227 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.369199 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls\") pod \"c07aacca-258f-4add-8dff-f66fa54231bb\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " Sep 15 20:55:24.369379 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.369259 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czjts\" (UniqueName: \"kubernetes.io/projected/c07aacca-258f-4add-8dff-f66fa54231bb-kube-api-access-czjts\") pod \"c07aacca-258f-4add-8dff-f66fa54231bb\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " Sep 15 20:55:24.369379 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.369290 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c07aacca-258f-4add-8dff-f66fa54231bb-openshift-service-ca\") pod \"c07aacca-258f-4add-8dff-f66fa54231bb\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " Sep 15 20:55:24.369497 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.369388 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-metrics-certs\") pod \"c07aacca-258f-4add-8dff-f66fa54231bb\" (UID: \"c07aacca-258f-4add-8dff-f66fa54231bb\") " Sep 15 20:55:24.369894 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.369865 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07aacca-258f-4add-8dff-f66fa54231bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c07aacca-258f-4add-8dff-f66fa54231bb" (UID: "c07aacca-258f-4add-8dff-f66fa54231bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:24.372627 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.372593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c07aacca-258f-4add-8dff-f66fa54231bb" (UID: "c07aacca-258f-4add-8dff-f66fa54231bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:55:24.372733 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.372603 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c07aacca-258f-4add-8dff-f66fa54231bb" (UID: "c07aacca-258f-4add-8dff-f66fa54231bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:55:24.372733 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.372686 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07aacca-258f-4add-8dff-f66fa54231bb-kube-api-access-czjts" (OuterVolumeSpecName: "kube-api-access-czjts") pod "c07aacca-258f-4add-8dff-f66fa54231bb" (UID: "c07aacca-258f-4add-8dff-f66fa54231bb"). InnerVolumeSpecName "kube-api-access-czjts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:55:24.471065 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.471009 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.471065 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.471052 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czjts\" (UniqueName: \"kubernetes.io/projected/c07aacca-258f-4add-8dff-f66fa54231bb-kube-api-access-czjts\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.471065 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.471069 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c07aacca-258f-4add-8dff-f66fa54231bb-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.471842 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.471082 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c07aacca-258f-4add-8dff-f66fa54231bb-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:55:24.638268 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.638178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:24.843298 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.843265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:24.977542 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.977452 2580 generic.go:356] "Generic (PLEG): container finished" podID="c07aacca-258f-4add-8dff-f66fa54231bb" containerID="9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6" exitCode=0 Sep 15 20:55:24.977542 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.977499 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" Sep 15 20:55:24.977542 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.977528 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" event={"ID":"c07aacca-258f-4add-8dff-f66fa54231bb","Type":"ContainerDied","Data":"9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6"} Sep 15 20:55:24.977792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.977593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw" event={"ID":"c07aacca-258f-4add-8dff-f66fa54231bb","Type":"ContainerDied","Data":"a98d25912d2df3de6a01200628595204b9aed7f0a5854fd01fce06f578f829f8"} Sep 15 20:55:24.977792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.977624 2580 scope.go:117] "RemoveContainer" containerID="9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6" Sep 15 20:55:24.988592 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.988570 2580 scope.go:117] "RemoveContainer" containerID="9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6" Sep 15 20:55:24.988882 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:55:24.988862 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6\": container with ID starting with 9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6 not found: ID does not exist" containerID="9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6" Sep 15 20:55:24.988950 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.988894 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6"} err="failed to get container status \"9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6\": rpc error: code = NotFound desc = could not find container \"9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6\": container with ID starting with 9b145ee1809d0f0063cf5411288b2ce36f19cc0aed15d4a6aa1e26460b817df6 not found: ID does not exist" Sep 15 20:55:24.999877 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:24.999845 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw"] Sep 15 20:55:25.006163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:25.006134 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e8af3fa9-644bb9f74-nsmdw"] Sep 15 20:55:25.202161 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:25.202125 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:25.315920 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:25.315883 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c07aacca-258f-4add-8dff-f66fa54231bb" path="/var/lib/kubelet/pods/c07aacca-258f-4add-8dff-f66fa54231bb/volumes" Sep 15 20:55:25.443572 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:25.443539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:25.543345 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:25.543303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:26.238859 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:26.238823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:26.859359 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:26.859326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:27.040519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:27.040477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:27.333926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:27.333888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:27.429585 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:27.429540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:27.785575 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:27.785531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:28.532398 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:28.532338 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:29.140073 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.140038 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm"] Sep 15 20:55:29.140955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.140921 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07aacca-258f-4add-8dff-f66fa54231bb" containerName="maas-api" Sep 15 20:55:29.140955 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.140948 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07aacca-258f-4add-8dff-f66fa54231bb" containerName="maas-api" Sep 15 20:55:29.141139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.141076 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07aacca-258f-4add-8dff-f66fa54231bb" containerName="maas-api" Sep 15 20:55:29.145801 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.145772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.148054 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.147876 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0143258d-serving-cert\"" Sep 15 20:55:29.155529 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.155491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm"] Sep 15 20:55:29.172086 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.172042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:29.235385 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.235288 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:29.321217 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.321182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-openshift-service-ca\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.321368 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.321229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.321368 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.321350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-metrics-certs\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.321481 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.321456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gc6\" (UniqueName: \"kubernetes.io/projected/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-kube-api-access-c4gc6\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.422954 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.422862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gc6\" (UniqueName: \"kubernetes.io/projected/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-kube-api-access-c4gc6\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.423145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.422981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-openshift-service-ca\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.423145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.423016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.423145 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.423086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-metrics-certs\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.423353 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:55:29.423170 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-serving-cert: secret "maas-api-e2e-ait-0143258d-serving-cert" not found Sep 15 20:55:29.423353 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:55:29.423229 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls podName:dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01 nodeName:}" failed. No retries permitted until 2026-09-15 20:55:29.923213541 +0000 UTC m=+1119.052950106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls") pod "maas-api-e2e-ait-0143258d-77d465479c-rn7bm" (UID: "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01") : secret "maas-api-e2e-ait-0143258d-serving-cert" not found Sep 15 20:55:29.424071 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.424037 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-openshift-service-ca\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.425705 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.425682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-metrics-certs\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.432010 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.431980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gc6\" (UniqueName: \"kubernetes.io/projected/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-kube-api-access-c4gc6\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.650262 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.650214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:29.735178 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.735097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:29.927750 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.927703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:29.930193 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:29.930160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls\") pod \"maas-api-e2e-ait-0143258d-77d465479c-rn7bm\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:30.033938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:30.033903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:30.061012 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:30.060975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:30.343916 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:30.343891 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm"] Sep 15 20:55:30.345278 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:55:30.345231 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfea7b0e_d6e0_48bd_9fa7_2be7c0bbef01.slice/crio-d54ae5f42b10233f0d2b97196c662d41c2c570220680d4f1c5828a4f79ffb693 WatchSource:0}: Error finding container d54ae5f42b10233f0d2b97196c662d41c2c570220680d4f1c5828a4f79ffb693: Status 404 returned error can't find the container with id d54ae5f42b10233f0d2b97196c662d41c2c570220680d4f1c5828a4f79ffb693 Sep 15 20:55:30.730239 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:30.730125 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:31.020220 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:31.020175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" event={"ID":"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01","Type":"ContainerStarted","Data":"d54ae5f42b10233f0d2b97196c662d41c2c570220680d4f1c5828a4f79ffb693"} Sep 15 20:55:31.407203 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:31.407169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:32.031951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:32.031908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" event={"ID":"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01","Type":"ContainerStarted","Data":"d1907e1ec00e5589d79b75e775f471582c66b42ca84e69a81fbc715422414313"} Sep 15 20:55:32.032346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:32.032079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:32.050730 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:32.050665 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" podStartSLOduration=2.024834985 podStartE2EDuration="3.050648572s" totalImagesPullingTime="1.025813587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:55:32.048398221 +0000 UTC m=+1121.178134829" watchObservedRunningTime="2026-09-15 20:55:32.050648572 +0000 UTC m=+1121.180385155" Sep 15 20:55:32.141975 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:32.141936 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:38.044777 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:38.044745 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:51.043579 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:51.043544 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:51.819738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:51.819694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:52.587473 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:52.587423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:53.223727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:53.223685 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:53.861475 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:53.861438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:54.259564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:54.259515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:54.750727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:54.750698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:55.427101 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:55.427069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:56.204540 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:56.204509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:56.841860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:56.841816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:57.517793 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:57.517752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:58.438858 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:58.438821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:58.981572 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:58.981531 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm"] Sep 15 20:55:58.981793 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:58.981769 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" podUID="dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" containerName="maas-api" containerID="cri-o://d1907e1ec00e5589d79b75e775f471582c66b42ca84e69a81fbc715422414313" gracePeriod=30 Sep 15 20:55:59.004812 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.004782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:59.100039 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.100004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:59.204808 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.204713 2580 generic.go:356] "Generic (PLEG): container finished" podID="dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" containerID="d1907e1ec00e5589d79b75e775f471582c66b42ca84e69a81fbc715422414313" exitCode=0 Sep 15 20:55:59.204808 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.204755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" event={"ID":"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01","Type":"ContainerDied","Data":"d1907e1ec00e5589d79b75e775f471582c66b42ca84e69a81fbc715422414313"} Sep 15 20:55:59.240548 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.240485 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:55:59.314538 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.314504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:59.344333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.344301 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls\") pod \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " Sep 15 20:55:59.344538 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.344393 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4gc6\" (UniqueName: \"kubernetes.io/projected/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-kube-api-access-c4gc6\") pod \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " Sep 15 20:55:59.344538 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.344437 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-metrics-certs\") pod \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " Sep 15 20:55:59.344538 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.344517 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-openshift-service-ca\") pod \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\" (UID: \"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01\") " Sep 15 20:55:59.344948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.344920 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" (UID: "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:55:59.346745 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.346718 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" (UID: "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:55:59.347045 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.347019 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" (UID: "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:55:59.347112 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.347058 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-kube-api-access-c4gc6" (OuterVolumeSpecName: "kube-api-access-c4gc6") pod "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" (UID: "dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01"). InnerVolumeSpecName "kube-api-access-c4gc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:55:59.369107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.369064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:55:59.446153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.446107 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gc6\" (UniqueName: \"kubernetes.io/projected/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-kube-api-access-c4gc6\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:55:59.446153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.446149 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:55:59.446153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.446164 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:55:59.446724 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:55:59.446177 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:00.036838 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.036804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:00.213346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.213305 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" event={"ID":"dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01","Type":"ContainerDied","Data":"d54ae5f42b10233f0d2b97196c662d41c2c570220680d4f1c5828a4f79ffb693"} Sep 15 20:56:00.213561 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.213385 2580 scope.go:117] "RemoveContainer" containerID="d1907e1ec00e5589d79b75e775f471582c66b42ca84e69a81fbc715422414313" Sep 15 20:56:00.213631 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.213579 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm" Sep 15 20:56:00.236007 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.235968 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm"] Sep 15 20:56:00.239309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.239280 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0143258d-77d465479c-rn7bm"] Sep 15 20:56:00.486608 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.486515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:00.816150 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.816107 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:00.955387 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:00.955340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:01.316989 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:01.316942 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" path="/var/lib/kubelet/pods/dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01/volumes" Sep 15 20:56:01.343034 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:01.342985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:01.708715 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:01.708629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:01.806711 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:01.806669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:02.532705 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:02.532670 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:03.531636 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:03.531591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:03.713977 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:03.713922 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:03.813741 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:03.813641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:04.375211 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:04.375174 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:05.142755 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:05.142714 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:05.936396 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:05.936312 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:06.594772 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:06.594736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:07.223156 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:07.223113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:07.824178 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:07.824140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:08.422593 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.422555 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt"] Sep 15 20:56:08.423329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.423304 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" containerName="maas-api" Sep 15 20:56:08.423329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.423327 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" containerName="maas-api" Sep 15 20:56:08.423556 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.423461 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfea7b0e-d6e0-48bd-9fa7-2be7c0bbef01" containerName="maas-api" Sep 15 20:56:08.429251 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.429223 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.431211 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.431187 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a7948686-serving-cert\"" Sep 15 20:56:08.439617 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.439584 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt"] Sep 15 20:56:08.455186 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.455156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:08.543473 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.543357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-metrics-certs\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.543681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.543568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bpl\" (UniqueName: \"kubernetes.io/projected/3b1c91cb-2069-46fc-a9d2-5a2263cab886-kube-api-access-l2bpl\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.543681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.543639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c91cb-2069-46fc-a9d2-5a2263cab886-openshift-service-ca\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.543681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.543661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.557274 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.557226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:08.645168 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.645126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bpl\" (UniqueName: \"kubernetes.io/projected/3b1c91cb-2069-46fc-a9d2-5a2263cab886-kube-api-access-l2bpl\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.645389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.645300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c91cb-2069-46fc-a9d2-5a2263cab886-openshift-service-ca\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.645389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.645343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.645582 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.645447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-metrics-certs\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.645582 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:56:08.645521 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-serving-cert: secret "maas-api-e2e-derive-a7948686-serving-cert" not found Sep 15 20:56:08.645663 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:56:08.645593 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls podName:3b1c91cb-2069-46fc-a9d2-5a2263cab886 nodeName:}" failed. No retries permitted until 2026-09-15 20:56:09.145576618 +0000 UTC m=+1158.275313179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls") pod "maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" (UID: "3b1c91cb-2069-46fc-a9d2-5a2263cab886") : secret "maas-api-e2e-derive-a7948686-serving-cert" not found Sep 15 20:56:08.646073 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.646054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c91cb-2069-46fc-a9d2-5a2263cab886-openshift-service-ca\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.647927 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.647908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-metrics-certs\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.662333 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.662303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bpl\" (UniqueName: \"kubernetes.io/projected/3b1c91cb-2069-46fc-a9d2-5a2263cab886-kube-api-access-l2bpl\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:08.737169 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.737057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:08.992463 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:08.992336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:09.101535 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:09.101498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:09.150277 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:09.150239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:09.152880 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:09.152851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls\") pod \"maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:09.343489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:09.343452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:09.354940 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:09.354911 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:09.622316 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:09.622283 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt"] Sep 15 20:56:09.623810 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:56:09.623776 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1c91cb_2069_46fc_a9d2_5a2263cab886.slice/crio-7f951b870dc2ccc04790855b1fe887c0f828dadfb5432db04fac881bf575a3d3 WatchSource:0}: Error finding container 7f951b870dc2ccc04790855b1fe887c0f828dadfb5432db04fac881bf575a3d3: Status 404 returned error can't find the container with id 7f951b870dc2ccc04790855b1fe887c0f828dadfb5432db04fac881bf575a3d3 Sep 15 20:56:10.016012 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:10.015977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:10.280902 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:10.280787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" event={"ID":"3b1c91cb-2069-46fc-a9d2-5a2263cab886","Type":"ContainerStarted","Data":"552a309d789bd0477774ca637691f79e83b69f4be4e97580ee5c4f9f32ce8887"} Sep 15 20:56:10.280902 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:10.280863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" event={"ID":"3b1c91cb-2069-46fc-a9d2-5a2263cab886","Type":"ContainerStarted","Data":"7f951b870dc2ccc04790855b1fe887c0f828dadfb5432db04fac881bf575a3d3"} Sep 15 20:56:10.280902 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:10.280895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:10.303808 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:10.299307 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" podStartSLOduration=1.875610979 podStartE2EDuration="2.299281221s" totalImagesPullingTime="423.670242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:56:10.295193022 +0000 UTC m=+1159.424929616" watchObservedRunningTime="2026-09-15 20:56:10.299281221 +0000 UTC m=+1159.429017805" Sep 15 20:56:12.983124 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:12.983086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:13.481997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.481955 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t"] Sep 15 20:56:13.482248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.482189 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" podUID="91a51588-f8bd-42e2-93c9-d88220bd7f59" containerName="maas-api" containerID="cri-o://d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b" gracePeriod=30 Sep 15 20:56:13.506194 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.506156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:13.570391 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.570351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:13.670019 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.669985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:13.750056 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.750029 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:56:13.793896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.791031 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:13.796788 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.796744 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-metrics-certs\") pod \"91a51588-f8bd-42e2-93c9-d88220bd7f59\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " Sep 15 20:56:13.796975 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.796869 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkhbs\" (UniqueName: \"kubernetes.io/projected/91a51588-f8bd-42e2-93c9-d88220bd7f59-kube-api-access-hkhbs\") pod \"91a51588-f8bd-42e2-93c9-d88220bd7f59\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " Sep 15 20:56:13.796975 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.796903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls\") pod \"91a51588-f8bd-42e2-93c9-d88220bd7f59\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " Sep 15 20:56:13.796975 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.796937 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a51588-f8bd-42e2-93c9-d88220bd7f59-openshift-service-ca\") pod \"91a51588-f8bd-42e2-93c9-d88220bd7f59\" (UID: \"91a51588-f8bd-42e2-93c9-d88220bd7f59\") " Sep 15 20:56:13.797734 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.797703 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a51588-f8bd-42e2-93c9-d88220bd7f59-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91a51588-f8bd-42e2-93c9-d88220bd7f59" (UID: "91a51588-f8bd-42e2-93c9-d88220bd7f59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:13.800142 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.800115 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a51588-f8bd-42e2-93c9-d88220bd7f59-kube-api-access-hkhbs" (OuterVolumeSpecName: "kube-api-access-hkhbs") pod "91a51588-f8bd-42e2-93c9-d88220bd7f59" (UID: "91a51588-f8bd-42e2-93c9-d88220bd7f59"). InnerVolumeSpecName "kube-api-access-hkhbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:13.800392 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.800369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91a51588-f8bd-42e2-93c9-d88220bd7f59" (UID: "91a51588-f8bd-42e2-93c9-d88220bd7f59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:56:13.801726 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.801701 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "91a51588-f8bd-42e2-93c9-d88220bd7f59" (UID: "91a51588-f8bd-42e2-93c9-d88220bd7f59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:56:13.898336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.898291 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhbs\" (UniqueName: \"kubernetes.io/projected/91a51588-f8bd-42e2-93c9-d88220bd7f59-kube-api-access-hkhbs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:13.898336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.898328 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:13.898336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.898342 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a51588-f8bd-42e2-93c9-d88220bd7f59-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:13.898632 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:13.898353 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a51588-f8bd-42e2-93c9-d88220bd7f59-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:14.140009 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.139970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:14.240538 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.240500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:14.308834 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.308799 2580 generic.go:356] "Generic (PLEG): container finished" podID="91a51588-f8bd-42e2-93c9-d88220bd7f59" containerID="d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b" exitCode=0 Sep 15 20:56:14.309028 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.308852 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" Sep 15 20:56:14.309028 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.308859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" event={"ID":"91a51588-f8bd-42e2-93c9-d88220bd7f59","Type":"ContainerDied","Data":"d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b"} Sep 15 20:56:14.309028 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.308898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t" event={"ID":"91a51588-f8bd-42e2-93c9-d88220bd7f59","Type":"ContainerDied","Data":"7bd4353b57dc52ae0a5d36cf1514080a2a8047a54f836ed79593b0c24a8be7f1"} Sep 15 20:56:14.309028 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.308922 2580 scope.go:117] "RemoveContainer" containerID="d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b" Sep 15 20:56:14.320548 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.320523 2580 scope.go:117] "RemoveContainer" containerID="d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b" Sep 15 20:56:14.320860 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:56:14.320838 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b\": container with ID starting with d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b not found: ID does not exist" containerID="d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b" Sep 15 20:56:14.320918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.320869 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b"} err="failed to get container status \"d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b\": rpc error: code = NotFound desc = could not find container \"d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b\": container with ID starting with d161c70e9af2fd0961b4a26343ef240d325e0ca5166c1186a84f488d4450280b not found: ID does not exist" Sep 15 20:56:14.334855 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.334825 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t"] Sep 15 20:56:14.341418 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.341377 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b9ff68-6c85ff974d-4l87t"] Sep 15 20:56:14.972741 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:14.972702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:15.316804 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:15.316764 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a51588-f8bd-42e2-93c9-d88220bd7f59" path="/var/lib/kubelet/pods/91a51588-f8bd-42e2-93c9-d88220bd7f59/volumes" Sep 15 20:56:15.637096 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:15.636996 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:16.293400 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:16.293371 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:16.421451 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:16.421416 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:17.076974 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:17.076935 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:17.794805 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:17.794770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:18.632594 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:18.632556 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:19.194181 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:19.194147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:19.394886 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:19.394850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:20.264492 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:20.264451 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:21.008089 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:21.008049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:21.835389 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:21.835330 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:22.549590 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:22.549551 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:26.055676 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:26.055643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:28.447121 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:28.447084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:29.141899 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.141846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:29.144297 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.144184 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt"] Sep 15 20:56:29.144757 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.144727 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" podUID="3b1c91cb-2069-46fc-a9d2-5a2263cab886" containerName="maas-api" containerID="cri-o://552a309d789bd0477774ca637691f79e83b69f4be4e97580ee5c4f9f32ce8887" gracePeriod=30 Sep 15 20:56:29.233801 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.233769 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:29.321168 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.321093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:29.375211 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.375169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:29.436949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.436595 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b1c91cb-2069-46fc-a9d2-5a2263cab886" containerID="552a309d789bd0477774ca637691f79e83b69f4be4e97580ee5c4f9f32ce8887" exitCode=0 Sep 15 20:56:29.436949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.436836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" event={"ID":"3b1c91cb-2069-46fc-a9d2-5a2263cab886","Type":"ContainerDied","Data":"552a309d789bd0477774ca637691f79e83b69f4be4e97580ee5c4f9f32ce8887"} Sep 15 20:56:29.436949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.436890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" event={"ID":"3b1c91cb-2069-46fc-a9d2-5a2263cab886","Type":"ContainerDied","Data":"7f951b870dc2ccc04790855b1fe887c0f828dadfb5432db04fac881bf575a3d3"} Sep 15 20:56:29.436949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.436916 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f951b870dc2ccc04790855b1fe887c0f828dadfb5432db04fac881bf575a3d3" Sep 15 20:56:29.459774 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.459689 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:29.492659 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.492604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:29.572042 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.572006 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2bpl\" (UniqueName: \"kubernetes.io/projected/3b1c91cb-2069-46fc-a9d2-5a2263cab886-kube-api-access-l2bpl\") pod \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " Sep 15 20:56:29.572218 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.572102 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls\") pod \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " Sep 15 20:56:29.572218 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.572168 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-metrics-certs\") pod \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " Sep 15 20:56:29.572218 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.572195 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c91cb-2069-46fc-a9d2-5a2263cab886-openshift-service-ca\") pod \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\" (UID: \"3b1c91cb-2069-46fc-a9d2-5a2263cab886\") " Sep 15 20:56:29.572658 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.572629 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1c91cb-2069-46fc-a9d2-5a2263cab886-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b1c91cb-2069-46fc-a9d2-5a2263cab886" (UID: "3b1c91cb-2069-46fc-a9d2-5a2263cab886"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:56:29.574276 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.574249 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b1c91cb-2069-46fc-a9d2-5a2263cab886" (UID: "3b1c91cb-2069-46fc-a9d2-5a2263cab886"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:56:29.574364 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.574313 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b1c91cb-2069-46fc-a9d2-5a2263cab886" (UID: "3b1c91cb-2069-46fc-a9d2-5a2263cab886"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:56:29.574364 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.574324 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1c91cb-2069-46fc-a9d2-5a2263cab886-kube-api-access-l2bpl" (OuterVolumeSpecName: "kube-api-access-l2bpl") pod "3b1c91cb-2069-46fc-a9d2-5a2263cab886" (UID: "3b1c91cb-2069-46fc-a9d2-5a2263cab886"). InnerVolumeSpecName "kube-api-access-l2bpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:56:29.590545 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.590504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:29.674024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.673930 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bpl\" (UniqueName: \"kubernetes.io/projected/3b1c91cb-2069-46fc-a9d2-5a2263cab886-kube-api-access-l2bpl\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:29.674024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.673962 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:29.674024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.673976 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c91cb-2069-46fc-a9d2-5a2263cab886-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:29.674024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.673990 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c91cb-2069-46fc-a9d2-5a2263cab886-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:56:29.735523 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:29.735482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:30.442481 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:30.442445 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt" Sep 15 20:56:30.484548 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:30.484503 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt"] Sep 15 20:56:30.486260 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:30.486232 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7948686-85b6cf7fcb-7jcvt"] Sep 15 20:56:30.608632 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:30.608591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:30.709302 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:30.709213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:31.317215 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:31.317167 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1c91cb-2069-46fc-a9d2-5a2263cab886" path="/var/lib/kubelet/pods/3b1c91cb-2069-46fc-a9d2-5a2263cab886/volumes" Sep 15 20:56:31.537958 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:31.537915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:31.986916 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:31.986877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:32.343463 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:32.343426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:33.144602 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:33.144565 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:38.347807 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:38.347773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:38.549442 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:38.549385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:38.708154 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:38.708074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:39.164248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:39.164209 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:56:51.350825 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:51.350797 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:56:51.355738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:51.355710 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 20:56:51.686966 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:56:51.686879 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:05.301446 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:05.301378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:07.310910 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:07.310865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:07.669327 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:07.669230 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f"] Sep 15 20:57:07.669534 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:07.669495 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" containerID="cri-o://438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613" gracePeriod=30 Sep 15 20:57:07.993498 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:07.993365 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:08.157475 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:08.157432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:08.580936 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:08.580903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:08.734138 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:08.734104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:11.105336 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:11.105292 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:11.706293 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:11.706245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:11.817742 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:11.817704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:12.240589 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:12.240558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:12.932980 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:12.932914 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:57:13.037774 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.037731 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istiod-ca-cert\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.037974 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.037787 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-certs\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.037974 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.037837 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-credential-socket\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.037974 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.037864 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-data\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.037974 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.037892 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-envoy\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.037974 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.037942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-token\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.038245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038003 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-socket\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.038245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038036 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:57:13.038245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038222 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:57:13.038245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038212 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:57:13.038245 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038229 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:57:13.038552 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038334 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:57:13.038552 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038367 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-podinfo\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.038552 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsdk7\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-kube-api-access-nsdk7\") pod \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\" (UID: \"cd53285b-d5c3-4c10-9979-72d35f95b2b7\") " Sep 15 20:57:13.038851 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038834 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.038938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038856 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.038938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038870 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.038938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038882 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.038938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.038895 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.040238 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.040216 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:57:13.040350 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.040257 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:57:13.040350 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.040297 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 20:57:13.040754 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.040730 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-kube-api-access-nsdk7" (OuterVolumeSpecName: "kube-api-access-nsdk7") pod "cd53285b-d5c3-4c10-9979-72d35f95b2b7" (UID: "cd53285b-d5c3-4c10-9979-72d35f95b2b7"). InnerVolumeSpecName "kube-api-access-nsdk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:57:13.139951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.139910 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.139951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.139940 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.139951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.139954 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd53285b-d5c3-4c10-9979-72d35f95b2b7-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.139951 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.139966 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdk7\" (UniqueName: \"kubernetes.io/projected/cd53285b-d5c3-4c10-9979-72d35f95b2b7-kube-api-access-nsdk7\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:57:13.200620 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.200514 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:13.305393 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.305347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:13.476340 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.476243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:13.723950 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.723910 2580 generic.go:356] "Generic (PLEG): container finished" podID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerID="438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613" exitCode=0 Sep 15 20:57:13.724153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.723960 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" Sep 15 20:57:13.724153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.723994 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" event={"ID":"cd53285b-d5c3-4c10-9979-72d35f95b2b7","Type":"ContainerDied","Data":"438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613"} Sep 15 20:57:13.724153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.724056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f" event={"ID":"cd53285b-d5c3-4c10-9979-72d35f95b2b7","Type":"ContainerDied","Data":"867b48228c291fc10524993a07683b99ebd7495e8c35449db72c70a1b848f2a1"} Sep 15 20:57:13.724153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.724088 2580 scope.go:117] "RemoveContainer" containerID="438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613" Sep 15 20:57:13.734107 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.734092 2580 scope.go:117] "RemoveContainer" containerID="b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135" Sep 15 20:57:13.742906 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.742887 2580 scope.go:117] "RemoveContainer" containerID="438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613" Sep 15 20:57:13.743219 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:57:13.743197 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613\": container with ID starting with 438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613 not found: ID does not exist" containerID="438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613" Sep 15 20:57:13.743286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.743228 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613"} err="failed to get container status \"438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613\": rpc error: code = NotFound desc = could not find container \"438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613\": container with ID starting with 438cf8c9f8a2e9003e0b26c16847bb7aa6f355f99230160c0f747d149f9bd613 not found: ID does not exist" Sep 15 20:57:13.743286 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.743248 2580 scope.go:117] "RemoveContainer" containerID="b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135" Sep 15 20:57:13.743564 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:57:13.743545 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135\": container with ID starting with b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135 not found: ID does not exist" containerID="b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135" Sep 15 20:57:13.743632 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.743575 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135"} err="failed to get container status \"b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135\": rpc error: code = NotFound desc = could not find container \"b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135\": container with ID starting with b39da2059660269c8d458315f81b1e18d5d328979fdfe8fb7a58dda513469135 not found: ID does not exist" Sep 15 20:57:13.748927 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.748900 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f"] Sep 15 20:57:13.751605 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:13.751584 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b9ff68-openshift-default-7756c45688-4dc5f"] Sep 15 20:57:14.407631 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:14.407585 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:15.135100 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.135066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:15.316906 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.316869 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" path="/var/lib/kubelet/pods/cd53285b-d5c3-4c10-9979-72d35f95b2b7/volumes" Sep 15 20:57:15.817595 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.817554 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn"] Sep 15 20:57:15.818085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818049 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a51588-f8bd-42e2-93c9-d88220bd7f59" containerName="maas-api" Sep 15 20:57:15.818085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818065 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a51588-f8bd-42e2-93c9-d88220bd7f59" containerName="maas-api" Sep 15 20:57:15.818085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818078 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" Sep 15 20:57:15.818085 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818086 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" Sep 15 20:57:15.818309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818122 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" Sep 15 20:57:15.818309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818131 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" Sep 15 20:57:15.818309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818144 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1c91cb-2069-46fc-a9d2-5a2263cab886" containerName="maas-api" Sep 15 20:57:15.818309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818152 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1c91cb-2069-46fc-a9d2-5a2263cab886" containerName="maas-api" Sep 15 20:57:15.818309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818260 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1c91cb-2069-46fc-a9d2-5a2263cab886" containerName="maas-api" Sep 15 20:57:15.818309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818269 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" Sep 15 20:57:15.818309 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.818276 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a51588-f8bd-42e2-93c9-d88220bd7f59" containerName="maas-api" Sep 15 20:57:15.822915 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.822892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.825137 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.825111 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-41588d-serving-cert\"" Sep 15 20:57:15.834196 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.834172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn"] Sep 15 20:57:15.852108 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.852072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:15.868326 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.868283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-metrics-certs\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.868533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.868334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248df42-51d1-4bbc-9f34-a66e12596668-openshift-service-ca\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.868533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.868473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7bwt\" (UniqueName: \"kubernetes.io/projected/e248df42-51d1-4bbc-9f34-a66e12596668-kube-api-access-c7bwt\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.868649 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.868536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.919430 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.919359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:15.969369 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.969327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.969606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.969461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-metrics-certs\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.969606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.969493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248df42-51d1-4bbc-9f34-a66e12596668-openshift-service-ca\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.969606 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:57:15.969530 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-serving-cert: secret "maas-api-e2e-explicit-41588d-serving-cert" not found Sep 15 20:57:15.969606 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.969571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7bwt\" (UniqueName: \"kubernetes.io/projected/e248df42-51d1-4bbc-9f34-a66e12596668-kube-api-access-c7bwt\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.969606 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:57:15.969599 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls podName:e248df42-51d1-4bbc-9f34-a66e12596668 nodeName:}" failed. No retries permitted until 2026-09-15 20:57:16.469576912 +0000 UTC m=+1225.599313482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls") pod "maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" (UID: "e248df42-51d1-4bbc-9f34-a66e12596668") : secret "maas-api-e2e-explicit-41588d-serving-cert" not found Sep 15 20:57:15.970319 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.970286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248df42-51d1-4bbc-9f34-a66e12596668-openshift-service-ca\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.972011 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.971987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-metrics-certs\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:15.982998 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:15.982975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7bwt\" (UniqueName: \"kubernetes.io/projected/e248df42-51d1-4bbc-9f34-a66e12596668-kube-api-access-c7bwt\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:16.043625 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:16.043584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:16.340367 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:16.340330 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:16.435143 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:16.435101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:16.475726 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:16.475685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:16.478139 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:16.478121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls\") pod \"maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:16.736993 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:16.736899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:17.014198 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:17.014164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn"] Sep 15 20:57:17.014824 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:57:17.014794 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode248df42_51d1_4bbc_9f34_a66e12596668.slice/crio-a349051ddfac1b661a8c80222ab452951560f15ee6606aa4e5a7175594c176b5 WatchSource:0}: Error finding container a349051ddfac1b661a8c80222ab452951560f15ee6606aa4e5a7175594c176b5: Status 404 returned error can't find the container with id a349051ddfac1b661a8c80222ab452951560f15ee6606aa4e5a7175594c176b5 Sep 15 20:57:17.752751 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:17.752716 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" event={"ID":"e248df42-51d1-4bbc-9f34-a66e12596668","Type":"ContainerStarted","Data":"f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f"} Sep 15 20:57:17.752915 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:17.752771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" event={"ID":"e248df42-51d1-4bbc-9f34-a66e12596668","Type":"ContainerStarted","Data":"a349051ddfac1b661a8c80222ab452951560f15ee6606aa4e5a7175594c176b5"} Sep 15 20:57:17.752915 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:17.752801 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:17.770354 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:17.770308 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" podStartSLOduration=2.203304126 podStartE2EDuration="2.770294415s" totalImagesPullingTime="566.990289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:57:17.768463972 +0000 UTC m=+1226.898200557" watchObservedRunningTime="2026-09-15 20:57:17.770294415 +0000 UTC m=+1226.900031002" Sep 15 20:57:23.733696 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:23.733662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:23.765556 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:23.765522 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:57:29.205861 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:29.205820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:35.881320 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:35.881278 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:35.987778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:35.987732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:36.092026 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:36.091982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:37.882929 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:37.882884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:38.755672 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:38.755632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:39.467579 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:39.467539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:40.153052 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:40.153015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:40.839397 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:40.839353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:41.535207 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:41.535171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:41.837790 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:41.837673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:46.669070 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:46.669035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:47.543359 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:47.543320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:49.781716 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.781676 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg"] Sep 15 20:57:49.782470 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.782448 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd53285b-d5c3-4c10-9979-72d35f95b2b7" containerName="istio-proxy" Sep 15 20:57:49.785870 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.785849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.788749 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.788723 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"openshift-service-ca.crt\"" Sep 15 20:57:49.789500 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.789479 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"default-dockercfg-qswjz\"" Sep 15 20:57:49.789608 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.789502 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"premium-w0-3f1c93-kserve-self-signed-certs\"" Sep 15 20:57:49.789608 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.789544 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"kube-root-ca.crt\"" Sep 15 20:57:49.795997 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.795971 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg"] Sep 15 20:57:49.817605 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.817574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tmp-dir\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.817798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.817635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rj6p\" (UniqueName: \"kubernetes.io/projected/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-kube-api-access-2rj6p\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.817798 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.817772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-home\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.817904 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.817801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tls-certs\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.817904 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.817826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-model-cache\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.817904 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.817851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-dshm\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.918700 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.918659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-dshm\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.918921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.918783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tmp-dir\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.919004 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.918924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rj6p\" (UniqueName: \"kubernetes.io/projected/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-kube-api-access-2rj6p\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.919316 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.919287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tmp-dir\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.919519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.919494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-home\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.919628 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.919540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tls-certs\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.919697 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.919654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-model-cache\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.919909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.919884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-home\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.920280 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.920248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-model-cache\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.922425 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.922378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-dshm\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.922642 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.922624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tls-certs\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.931002 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.930969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rj6p\" (UniqueName: \"kubernetes.io/projected/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-kube-api-access-2rj6p\") pod \"premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:49.970870 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:49.970813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:50.096672 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:50.096577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:57:50.377222 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:50.377194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg"] Sep 15 20:57:50.378893 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:57:50.378865 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f9bcad3_3d3c_46c9_a855_1b739d78cf8c.slice/crio-ee90c0043557a8ad32f27e61f91fd1562606fa5e99049e39733b22fd65321041 WatchSource:0}: Error finding container ee90c0043557a8ad32f27e61f91fd1562606fa5e99049e39733b22fd65321041: Status 404 returned error can't find the container with id ee90c0043557a8ad32f27e61f91fd1562606fa5e99049e39733b22fd65321041 Sep 15 20:57:50.957487 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:50.957451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" event={"ID":"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c","Type":"ContainerStarted","Data":"c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc"} Sep 15 20:57:50.957949 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:50.957496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" event={"ID":"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c","Type":"ContainerStarted","Data":"ee90c0043557a8ad32f27e61f91fd1562606fa5e99049e39733b22fd65321041"} Sep 15 20:57:50.978718 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:50.978649 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" podStartSLOduration=1.978630262 podStartE2EDuration="1.978630262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:57:50.977668022 +0000 UTC m=+1260.107404608" watchObservedRunningTime="2026-09-15 20:57:50.978630262 +0000 UTC m=+1260.108366844" Sep 15 20:57:53.589278 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:53.589235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:53.696994 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:53.696962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:53.869572 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:53.869485 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:53.957830 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:53.957798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:54.637180 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:54.637147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:54.826064 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:54.826029 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:56.409935 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:56.409899 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:57:59.081103 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:57:59.081058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:00.096840 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:00.096798 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:58:00.097308 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:00.096856 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:58:00.109307 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:00.109276 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:58:01.034082 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:01.034047 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 20:58:02.494428 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:02.494336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:02.598022 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:02.597991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:03.189111 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:03.189069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:08.884442 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:08.884389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:08.985996 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:08.985956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:09.152964 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.152874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:09.438960 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.438878 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q"] Sep 15 20:58:09.442930 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.442906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.445463 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.445440 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c103b8e8-openshift-default-dockercfg-zs4pm\"" Sep 15 20:58:09.463189 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.463158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q"] Sep 15 20:58:09.485635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.485600 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:09.629782 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.629744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2xd\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-kube-api-access-hw2xd\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.629948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.629792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-istio-token\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.629948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.629876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-credential-socket\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.629948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.629901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6168c6b-6f80-4863-9711-b950e832de5d-istiod-ca-cert\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.629948 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.629919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-envoy\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.630094 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.630034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-data\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.630094 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.630078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-certs\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.630160 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.630124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-socket\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.630160 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.630140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6168c6b-6f80-4863-9711-b950e832de5d-istio-podinfo\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.666967 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.666928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:09.730860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.730770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-data\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.730860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.730813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-certs\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.730860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.730846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-socket\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.730860 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.730861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6168c6b-6f80-4863-9711-b950e832de5d-istio-podinfo\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.730896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2xd\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-kube-api-access-hw2xd\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.730932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-istio-token\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-credential-socket\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6168c6b-6f80-4863-9711-b950e832de5d-istiod-ca-cert\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731206 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-envoy\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731523 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-socket\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731523 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-data\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731634 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-credential-socket\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731856 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6168c6b-6f80-4863-9711-b950e832de5d-istiod-ca-cert\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.731972 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.731940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-certs\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.733232 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.733210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-envoy\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.733443 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.733423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6168c6b-6f80-4863-9711-b950e832de5d-istio-podinfo\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.743243 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.743214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-istio-token\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.744158 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.744138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2xd\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-kube-api-access-hw2xd\") pod \"e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:09.754086 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:09.754067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:10.037600 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:10.037572 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q"] Sep 15 20:58:10.039831 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:58:10.039799 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6168c6b_6f80_4863_9711_b950e832de5d.slice/crio-013718c5126a908bd3d1cf73b5a5acd6cf9e1cde16ab2cf971e2e7083cee645a WatchSource:0}: Error finding container 013718c5126a908bd3d1cf73b5a5acd6cf9e1cde16ab2cf971e2e7083cee645a: Status 404 returned error can't find the container with id 013718c5126a908bd3d1cf73b5a5acd6cf9e1cde16ab2cf971e2e7083cee645a Sep 15 20:58:10.042144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:10.042110 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:58:10.042234 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:10.042176 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:58:10.042234 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:10.042206 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:58:10.080466 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:10.080432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" event={"ID":"a6168c6b-6f80-4863-9711-b950e832de5d","Type":"ContainerStarted","Data":"013718c5126a908bd3d1cf73b5a5acd6cf9e1cde16ab2cf971e2e7083cee645a"} Sep 15 20:58:11.088271 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:11.088235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" event={"ID":"a6168c6b-6f80-4863-9711-b950e832de5d","Type":"ContainerStarted","Data":"04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0"} Sep 15 20:58:11.115983 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:11.115918 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podStartSLOduration=2.115895386 podStartE2EDuration="2.115895386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:11.109553051 +0000 UTC m=+1280.239289633" watchObservedRunningTime="2026-09-15 20:58:11.115895386 +0000 UTC m=+1280.245631970" Sep 15 20:58:11.754470 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:11.754429 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:11.759515 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:11.759490 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:12.094957 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:12.094877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:12.095931 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:12.095911 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:14.623419 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:14.623373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:15.275280 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:15.275240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:15.365447 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:15.365391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:16.117505 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:16.117464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:16.825313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:16.825265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:17.567579 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:17.567539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:18.220978 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:18.220946 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:18.917932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:18.917900 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws"] Sep 15 20:58:18.922754 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:18.922728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:18.924871 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:18.924849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c103b8e8-serving-cert\"" Sep 15 20:58:18.929931 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:18.929906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws"] Sep 15 20:58:18.944526 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:18.944498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:19.028759 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.028658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a033550-68e2-489c-b7d9-bfa29433c2b8-openshift-service-ca\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.028759 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.028723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-metrics-certs\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.028759 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.028765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-maas-api-tls\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.029733 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.028884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2828p\" (UniqueName: \"kubernetes.io/projected/6a033550-68e2-489c-b7d9-bfa29433c2b8-kube-api-access-2828p\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.045713 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.045677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:19.130160 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.130129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-metrics-certs\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.130315 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.130170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-maas-api-tls\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.130315 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.130253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2828p\" (UniqueName: \"kubernetes.io/projected/6a033550-68e2-489c-b7d9-bfa29433c2b8-kube-api-access-2828p\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.130480 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.130325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a033550-68e2-489c-b7d9-bfa29433c2b8-openshift-service-ca\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.130926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.130905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a033550-68e2-489c-b7d9-bfa29433c2b8-openshift-service-ca\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.132695 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.132677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-metrics-certs\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.132755 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.132717 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-maas-api-tls\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.139379 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.139351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2828p\" (UniqueName: \"kubernetes.io/projected/6a033550-68e2-489c-b7d9-bfa29433c2b8-kube-api-access-2828p\") pod \"maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.146163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.146137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:19.237118 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.237031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:19.402130 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.402064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:19.503393 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.503358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:19.518422 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:19.518381 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws"] Sep 15 20:58:19.520265 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:58:19.520240 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a033550_68e2_489c_b7d9_bfa29433c2b8.slice/crio-81b322290764a40e37787526720d295fe5ee743edcb159f211beb14c6fcb4624 WatchSource:0}: Error finding container 81b322290764a40e37787526720d295fe5ee743edcb159f211beb14c6fcb4624: Status 404 returned error can't find the container with id 81b322290764a40e37787526720d295fe5ee743edcb159f211beb14c6fcb4624 Sep 15 20:58:20.147483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:20.147437 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" event={"ID":"6a033550-68e2-489c-b7d9-bfa29433c2b8","Type":"ContainerStarted","Data":"2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1"} Sep 15 20:58:20.147895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:20.147508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" event={"ID":"6a033550-68e2-489c-b7d9-bfa29433c2b8","Type":"ContainerStarted","Data":"81b322290764a40e37787526720d295fe5ee743edcb159f211beb14c6fcb4624"} Sep 15 20:58:20.147895 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:20.147579 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:20.188729 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:20.188677 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" podStartSLOduration=1.6866212649999999 podStartE2EDuration="2.188658404s" totalImagesPullingTime="502.037139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:20.186604302 +0000 UTC m=+1289.316340875" watchObservedRunningTime="2026-09-15 20:58:20.188658404 +0000 UTC m=+1289.318394987" Sep 15 20:58:26.159650 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:26.159618 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:58:40.928199 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:40.928160 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:41.289111 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:41.289079 2580 generic.go:356] "Generic (PLEG): container finished" podID="a6168c6b-6f80-4863-9711-b950e832de5d" containerID="04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0" exitCode=0 Sep 15 20:58:41.289284 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:41.289131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" event={"ID":"a6168c6b-6f80-4863-9711-b950e832de5d","Type":"ContainerDied","Data":"04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0"} Sep 15 20:58:41.289568 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:41.289550 2580 scope.go:117] "RemoveContainer" containerID="04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0" Sep 15 20:58:41.290740 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:41.290712 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:58:41.290802 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:41.290789 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:58:41.290838 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:41.290822 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:58:41.732510 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:41.732394 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:42.095669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:42.095630 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:42.297826 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:42.297791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" event={"ID":"a6168c6b-6f80-4863-9711-b950e832de5d","Type":"ContainerStarted","Data":"f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377"} Sep 15 20:58:42.432172 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:42.432069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:42.754513 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:42.754474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:42.755846 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:42.755822 2580 patch_prober.go:28] interesting pod/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 15 20:58:42.755919 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:42.755874 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 15 20:58:43.094884 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:43.094799 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:43.755329 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:43.755290 2580 patch_prober.go:28] interesting pod/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 15 20:58:43.755747 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:43.755356 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 15 20:58:43.824672 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:43.824629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:44.449737 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:44.449706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:44.755293 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:44.755261 2580 patch_prober.go:28] interesting pod/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 15 20:58:44.755483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:44.755316 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 15 20:58:45.222468 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:45.222347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:45.537034 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:45.536993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:45.755521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:45.755482 2580 patch_prober.go:28] interesting pod/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 15 20:58:45.755918 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:45.755543 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 15 20:58:46.251218 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:46.251179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:46.351017 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:46.350977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:46.729897 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:46.729785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:46.754635 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:46.754598 2580 patch_prober.go:28] interesting pod/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 15 20:58:46.754810 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:46.754666 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 15 20:58:47.755707 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:47.755659 2580 patch_prober.go:28] interesting pod/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 15 20:58:47.756135 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:47.755737 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 15 20:58:48.758521 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:48.758489 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:48.758999 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:48.758795 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:48.759590 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:48.759568 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:58:52.477512 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:52.477476 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:52.540321 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:52.540287 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:53.598628 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:53.598588 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:54.175328 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:54.175290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:54.460569 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:54.460471 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:55.077984 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:55.077948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:55.825320 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:55.825274 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:56.454738 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:56.454707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:57.223894 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:57.223849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:58:57.849529 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:58:57.849498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:03.365673 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.365640 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws"] Sep 15 20:59:03.366063 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.365850 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" podUID="6a033550-68e2-489c-b7d9-bfa29433c2b8" containerName="maas-api" containerID="cri-o://2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1" gracePeriod=30 Sep 15 20:59:03.441204 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.441163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:03.542565 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.542465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:03.618774 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.618710 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:59:03.641367 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.641332 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:03.729770 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.729738 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-maas-api-tls\") pod \"6a033550-68e2-489c-b7d9-bfa29433c2b8\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " Sep 15 20:59:03.729965 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.729813 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2828p\" (UniqueName: \"kubernetes.io/projected/6a033550-68e2-489c-b7d9-bfa29433c2b8-kube-api-access-2828p\") pod \"6a033550-68e2-489c-b7d9-bfa29433c2b8\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " Sep 15 20:59:03.729965 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.729835 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-metrics-certs\") pod \"6a033550-68e2-489c-b7d9-bfa29433c2b8\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " Sep 15 20:59:03.729965 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.729855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a033550-68e2-489c-b7d9-bfa29433c2b8-openshift-service-ca\") pod \"6a033550-68e2-489c-b7d9-bfa29433c2b8\" (UID: \"6a033550-68e2-489c-b7d9-bfa29433c2b8\") " Sep 15 20:59:03.730347 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.730313 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a033550-68e2-489c-b7d9-bfa29433c2b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a033550-68e2-489c-b7d9-bfa29433c2b8" (UID: "6a033550-68e2-489c-b7d9-bfa29433c2b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:59:03.732104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.732070 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a033550-68e2-489c-b7d9-bfa29433c2b8" (UID: "6a033550-68e2-489c-b7d9-bfa29433c2b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:59:03.732197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.732114 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a033550-68e2-489c-b7d9-bfa29433c2b8" (UID: "6a033550-68e2-489c-b7d9-bfa29433c2b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:59:03.732197 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.732136 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a033550-68e2-489c-b7d9-bfa29433c2b8-kube-api-access-2828p" (OuterVolumeSpecName: "kube-api-access-2828p") pod "6a033550-68e2-489c-b7d9-bfa29433c2b8" (UID: "6a033550-68e2-489c-b7d9-bfa29433c2b8"). InnerVolumeSpecName "kube-api-access-2828p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:03.831018 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.830981 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:03.831018 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.831012 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2828p\" (UniqueName: \"kubernetes.io/projected/6a033550-68e2-489c-b7d9-bfa29433c2b8-kube-api-access-2828p\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:03.831018 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.831023 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a033550-68e2-489c-b7d9-bfa29433c2b8-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:03.831304 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.831031 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a033550-68e2-489c-b7d9-bfa29433c2b8-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:03.944568 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:03.944486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:04.044754 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.044695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:04.459187 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.459152 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a033550-68e2-489c-b7d9-bfa29433c2b8" containerID="2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1" exitCode=0 Sep 15 20:59:04.459622 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.459192 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" Sep 15 20:59:04.459622 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.459236 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" event={"ID":"6a033550-68e2-489c-b7d9-bfa29433c2b8","Type":"ContainerDied","Data":"2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1"} Sep 15 20:59:04.459622 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.459298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws" event={"ID":"6a033550-68e2-489c-b7d9-bfa29433c2b8","Type":"ContainerDied","Data":"81b322290764a40e37787526720d295fe5ee743edcb159f211beb14c6fcb4624"} Sep 15 20:59:04.459622 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.459318 2580 scope.go:117] "RemoveContainer" containerID="2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1" Sep 15 20:59:04.472717 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.472697 2580 scope.go:117] "RemoveContainer" containerID="2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1" Sep 15 20:59:04.473176 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:59:04.473153 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1\": container with ID starting with 2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1 not found: ID does not exist" containerID="2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1" Sep 15 20:59:04.473261 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.473186 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1"} err="failed to get container status \"2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1\": rpc error: code = NotFound desc = could not find container \"2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1\": container with ID starting with 2d623dbcbdaefc7381deb308a1531aa25a0308433dcff5478660786c313b08d1 not found: ID does not exist" Sep 15 20:59:04.484399 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.484372 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws"] Sep 15 20:59:04.488098 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.488076 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c103b8e8-6ff6699c5-gpqws"] Sep 15 20:59:04.699326 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:04.699292 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:05.316028 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:05.315979 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a033550-68e2-489c-b7d9-bfa29433c2b8" path="/var/lib/kubelet/pods/6a033550-68e2-489c-b7d9-bfa29433c2b8/volumes" Sep 15 20:59:05.439555 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:05.439518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:06.240984 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:06.240948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:06.895616 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:06.895579 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:07.674616 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:07.674578 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:09.832338 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:09.832304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:09.899686 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:09.899645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:10.095686 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:10.095598 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q"] Sep 15 20:59:10.095896 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:10.095869 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" containerID="cri-o://f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377" gracePeriod=30 Sep 15 20:59:10.104577 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:10.104542 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:10.478103 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:10.478007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:10.572432 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:10.572377 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:10.722693 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:10.722650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:15.362623 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.362552 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:59:15.452553 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452506 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6168c6b-6f80-4863-9711-b950e832de5d-istiod-ca-cert\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.452765 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452563 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-socket\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.452765 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452634 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-envoy\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.452765 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452678 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-credential-socket\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.452765 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452713 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-certs\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.452765 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-data\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.453042 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-istio-token\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.453042 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw2xd\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-kube-api-access-hw2xd\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.453042 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452960 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6168c6b-6f80-4863-9711-b950e832de5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:59:15.453042 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452983 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:15.453042 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453002 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:15.453318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453057 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6168c6b-6f80-4863-9711-b950e832de5d-istio-podinfo\") pod \"a6168c6b-6f80-4863-9711-b950e832de5d\" (UID: \"a6168c6b-6f80-4863-9711-b950e832de5d\") " Sep 15 20:59:15.453318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453084 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:15.453318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.452852 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:15.453691 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453664 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.453802 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453695 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.453802 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453709 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6168c6b-6f80-4863-9711-b950e832de5d-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.453802 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453722 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.453802 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.453735 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.455183 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.455154 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 20:59:15.455183 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.455172 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-kube-api-access-hw2xd" (OuterVolumeSpecName: "kube-api-access-hw2xd") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "kube-api-access-hw2xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:15.455363 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.455342 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:15.455448 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.455340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6168c6b-6f80-4863-9711-b950e832de5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6168c6b-6f80-4863-9711-b950e832de5d" (UID: "a6168c6b-6f80-4863-9711-b950e832de5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 20:59:15.534259 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.534221 2580 generic.go:356] "Generic (PLEG): container finished" podID="a6168c6b-6f80-4863-9711-b950e832de5d" containerID="f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377" exitCode=0 Sep 15 20:59:15.534259 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.534268 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" Sep 15 20:59:15.534499 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.534302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" event={"ID":"a6168c6b-6f80-4863-9711-b950e832de5d","Type":"ContainerDied","Data":"f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377"} Sep 15 20:59:15.534499 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.534387 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q" event={"ID":"a6168c6b-6f80-4863-9711-b950e832de5d","Type":"ContainerDied","Data":"013718c5126a908bd3d1cf73b5a5acd6cf9e1cde16ab2cf971e2e7083cee645a"} Sep 15 20:59:15.534499 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.534423 2580 scope.go:117] "RemoveContainer" containerID="f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377" Sep 15 20:59:15.545855 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.545831 2580 scope.go:117] "RemoveContainer" containerID="04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0" Sep 15 20:59:15.554457 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.554432 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.554574 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.554459 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw2xd\" (UniqueName: \"kubernetes.io/projected/a6168c6b-6f80-4863-9711-b950e832de5d-kube-api-access-hw2xd\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.554574 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.554470 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6168c6b-6f80-4863-9711-b950e832de5d-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.554574 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.554480 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6168c6b-6f80-4863-9711-b950e832de5d-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:15.555317 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.555305 2580 scope.go:117] "RemoveContainer" containerID="f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377" Sep 15 20:59:15.555718 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:59:15.555691 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377\": container with ID starting with f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377 not found: ID does not exist" containerID="f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377" Sep 15 20:59:15.555818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.555724 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377"} err="failed to get container status \"f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377\": rpc error: code = NotFound desc = could not find container \"f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377\": container with ID starting with f00a1876a8fc1b5b3de9037b0c5b8ecb9fcf0b157b981a77795c30e31fe66377 not found: ID does not exist" Sep 15 20:59:15.555818 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.555742 2580 scope.go:117] "RemoveContainer" containerID="04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0" Sep 15 20:59:15.556025 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:59:15.556008 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0\": container with ID starting with 04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0 not found: ID does not exist" containerID="04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0" Sep 15 20:59:15.556081 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.556029 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0"} err="failed to get container status \"04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0\": rpc error: code = NotFound desc = could not find container \"04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0\": container with ID starting with 04bddb68ec350941bafc5a31734329fda5055a115ac338f6cbc021010ef682c0 not found: ID does not exist" Sep 15 20:59:15.557727 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.557681 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q"] Sep 15 20:59:15.560731 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:15.560711 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c103b8e8-openshift-default-67cc465b4-kp62q"] Sep 15 20:59:16.500926 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:16.500890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:16.603468 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:16.603435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:16.752098 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:16.752008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:16.868305 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:16.868272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:17.316104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:17.316064 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" path="/var/lib/kubelet/pods/a6168c6b-6f80-4863-9711-b950e832de5d/volumes" Sep 15 20:59:18.690891 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:18.690856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:19.060564 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:19.060523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:19.739528 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:19.739492 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:20.548714 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:20.548675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:21.205724 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:21.205686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:21.941357 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:21.941323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:22.559353 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:22.559313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:23.012036 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:23.011992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:23.112618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:23.112573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:23.230250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:23.230215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:23.962884 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:23.962845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:24.262029 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.261988 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75"] Sep 15 20:59:24.262523 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.262506 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" Sep 15 20:59:24.262523 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.262523 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" Sep 15 20:59:24.262678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.262551 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a033550-68e2-489c-b7d9-bfa29433c2b8" containerName="maas-api" Sep 15 20:59:24.262678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.262557 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a033550-68e2-489c-b7d9-bfa29433c2b8" containerName="maas-api" Sep 15 20:59:24.262678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.262625 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" Sep 15 20:59:24.262678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.262637 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a033550-68e2-489c-b7d9-bfa29433c2b8" containerName="maas-api" Sep 15 20:59:24.265888 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.265866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.268100 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.268078 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-de4215-openshift-default-dockercfg-dsn6n\"" Sep 15 20:59:24.278897 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.278863 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75"] Sep 15 20:59:24.288303 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.288275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:24.340250 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istiod-ca-cert\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340485 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-data\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340485 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-certs\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340485 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340362 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fsdf\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-kube-api-access-2fsdf\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340485 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-credential-socket\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340485 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-token\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-socket\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-podinfo\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.340678 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.340571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-envoy\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.441591 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istiod-ca-cert\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.441821 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-data\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.441821 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-certs\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.441821 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsdf\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-kube-api-access-2fsdf\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.441821 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-credential-socket\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.441821 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-token\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442099 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-socket\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442099 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-podinfo\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442099 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.441917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-envoy\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442267 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.442099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-certs\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442267 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.442142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-data\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442267 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.442219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-credential-socket\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442420 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.442319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-socket\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.442420 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.442377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istiod-ca-cert\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.444144 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.444120 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-envoy\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.444383 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.444365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-podinfo\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.449163 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.449142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-token\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.449334 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.449316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsdf\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-kube-api-access-2fsdf\") pod \"e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.578099 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.577996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:24.653434 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.652679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:24.864457 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.864427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75"] Sep 15 20:59:24.866790 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:59:24.866754 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0817b405_ca0c_4ec7_bdd0_71dc6337f7ce.slice/crio-96b58c2f0969a7cf8029f98b36764903ba4cd15c7cd3bff062a9d4e352ac74d2 WatchSource:0}: Error finding container 96b58c2f0969a7cf8029f98b36764903ba4cd15c7cd3bff062a9d4e352ac74d2: Status 404 returned error can't find the container with id 96b58c2f0969a7cf8029f98b36764903ba4cd15c7cd3bff062a9d4e352ac74d2 Sep 15 20:59:24.869627 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.869584 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:24.869725 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.869677 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:24.869778 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:24.869724 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:25.604651 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:25.604601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" event={"ID":"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce","Type":"ContainerStarted","Data":"49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6"} Sep 15 20:59:25.605072 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:25.604683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" event={"ID":"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce","Type":"ContainerStarted","Data":"96b58c2f0969a7cf8029f98b36764903ba4cd15c7cd3bff062a9d4e352ac74d2"} Sep 15 20:59:25.624312 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:25.624257 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" podStartSLOduration=1.6242395570000001 podStartE2EDuration="1.624239557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:25.621719463 +0000 UTC m=+1354.751456071" watchObservedRunningTime="2026-09-15 20:59:25.624239557 +0000 UTC m=+1354.753976139" Sep 15 20:59:26.579061 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:26.579026 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:26.584492 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:26.584468 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:26.611099 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:26.611067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:26.612337 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:26.612307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:26.837938 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:26.837855 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:27.645986 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:27.645952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:27.733225 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:27.733179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:28.370432 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:28.370376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:29.135154 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.135120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:29.686489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.686454 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn"] Sep 15 20:59:29.686701 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.686678 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" podUID="e248df42-51d1-4bbc-9f34-a66e12596668" containerName="maas-api" containerID="cri-o://f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f" gracePeriod=30 Sep 15 20:59:29.703852 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.703810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:29.803529 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.803493 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:29.895215 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.895184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:29.953483 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.953392 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:59:29.956551 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:29.956523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:30.003890 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.003857 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248df42-51d1-4bbc-9f34-a66e12596668-openshift-service-ca\") pod \"e248df42-51d1-4bbc-9f34-a66e12596668\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " Sep 15 20:59:30.004091 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.003972 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls\") pod \"e248df42-51d1-4bbc-9f34-a66e12596668\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " Sep 15 20:59:30.004091 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.004005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7bwt\" (UniqueName: \"kubernetes.io/projected/e248df42-51d1-4bbc-9f34-a66e12596668-kube-api-access-c7bwt\") pod \"e248df42-51d1-4bbc-9f34-a66e12596668\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " Sep 15 20:59:30.004091 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.004036 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-metrics-certs\") pod \"e248df42-51d1-4bbc-9f34-a66e12596668\" (UID: \"e248df42-51d1-4bbc-9f34-a66e12596668\") " Sep 15 20:59:30.004265 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.004226 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e248df42-51d1-4bbc-9f34-a66e12596668-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e248df42-51d1-4bbc-9f34-a66e12596668" (UID: "e248df42-51d1-4bbc-9f34-a66e12596668"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 20:59:30.004397 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.004373 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248df42-51d1-4bbc-9f34-a66e12596668-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:30.006256 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.006222 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e248df42-51d1-4bbc-9f34-a66e12596668-kube-api-access-c7bwt" (OuterVolumeSpecName: "kube-api-access-c7bwt") pod "e248df42-51d1-4bbc-9f34-a66e12596668" (UID: "e248df42-51d1-4bbc-9f34-a66e12596668"). InnerVolumeSpecName "kube-api-access-c7bwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 20:59:30.006256 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.006242 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e248df42-51d1-4bbc-9f34-a66e12596668" (UID: "e248df42-51d1-4bbc-9f34-a66e12596668"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:59:30.006433 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.006226 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e248df42-51d1-4bbc-9f34-a66e12596668" (UID: "e248df42-51d1-4bbc-9f34-a66e12596668"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 20:59:30.075909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.075874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:30.106207 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.106120 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:30.106207 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.106164 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7bwt\" (UniqueName: \"kubernetes.io/projected/e248df42-51d1-4bbc-9f34-a66e12596668-kube-api-access-c7bwt\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:30.106207 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.106179 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e248df42-51d1-4bbc-9f34-a66e12596668-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 20:59:30.249116 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.249027 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:30.646026 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.645989 2580 generic.go:356] "Generic (PLEG): container finished" podID="e248df42-51d1-4bbc-9f34-a66e12596668" containerID="f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f" exitCode=0 Sep 15 20:59:30.646225 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.646051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" event={"ID":"e248df42-51d1-4bbc-9f34-a66e12596668","Type":"ContainerDied","Data":"f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f"} Sep 15 20:59:30.646225 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.646061 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" Sep 15 20:59:30.646225 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.646116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn" event={"ID":"e248df42-51d1-4bbc-9f34-a66e12596668","Type":"ContainerDied","Data":"a349051ddfac1b661a8c80222ab452951560f15ee6606aa4e5a7175594c176b5"} Sep 15 20:59:30.646225 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.646137 2580 scope.go:117] "RemoveContainer" containerID="f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f" Sep 15 20:59:30.657995 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.657975 2580 scope.go:117] "RemoveContainer" containerID="f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f" Sep 15 20:59:30.658290 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:59:30.658268 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f\": container with ID starting with f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f not found: ID does not exist" containerID="f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f" Sep 15 20:59:30.658340 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.658299 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f"} err="failed to get container status \"f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f\": rpc error: code = NotFound desc = could not find container \"f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f\": container with ID starting with f5c578afc49a0c2481794b0111bf0b7bf4008a193e5c50d30c0ff78801833b7f not found: ID does not exist" Sep 15 20:59:30.671489 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.671458 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn"] Sep 15 20:59:30.677249 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.677217 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-41588d-7fbdc66d54-6htwn"] Sep 15 20:59:30.744153 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.744119 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q"] Sep 15 20:59:30.744804 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.744777 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e248df42-51d1-4bbc-9f34-a66e12596668" containerName="maas-api" Sep 15 20:59:30.744804 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.744802 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e248df42-51d1-4bbc-9f34-a66e12596668" containerName="maas-api" Sep 15 20:59:30.745023 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.744838 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" Sep 15 20:59:30.745023 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.744849 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" Sep 15 20:59:30.745023 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.744966 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e248df42-51d1-4bbc-9f34-a66e12596668" containerName="maas-api" Sep 15 20:59:30.745023 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.744980 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6168c6b-6f80-4863-9711-b950e832de5d" containerName="istio-proxy" Sep 15 20:59:30.749819 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.749797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.752299 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.752277 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-de4215-serving-cert\"" Sep 15 20:59:30.759255 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.759224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q"] Sep 15 20:59:30.790080 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.790047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:30.814104 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.814054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxz26\" (UniqueName: \"kubernetes.io/projected/7e1af04f-1277-4565-8f0b-344bce74c765-kube-api-access-gxz26\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.814312 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.814126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.814312 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.814159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1af04f-1277-4565-8f0b-344bce74c765-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.814312 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.814256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.889792 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.889756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:30.915170 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.915068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxz26\" (UniqueName: \"kubernetes.io/projected/7e1af04f-1277-4565-8f0b-344bce74c765-kube-api-access-gxz26\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.915170 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.915139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.915457 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.915243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1af04f-1277-4565-8f0b-344bce74c765-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.915457 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:59:30.915258 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-serving-cert: secret "maas-api-e2e-shared-a-w2-de4215-serving-cert" not found Sep 15 20:59:30.915457 ip-10-0-139-201 kubenswrapper[2580]: E0915 20:59:30.915335 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls podName:7e1af04f-1277-4565-8f0b-344bce74c765 nodeName:}" failed. No retries permitted until 2026-09-15 20:59:31.415312364 +0000 UTC m=+1360.545048939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls") pod "maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" (UID: "7e1af04f-1277-4565-8f0b-344bce74c765") : secret "maas-api-e2e-shared-a-w2-de4215-serving-cert" not found Sep 15 20:59:30.915457 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.915383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.916137 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.916112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1af04f-1277-4565-8f0b-344bce74c765-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.917864 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.917844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:30.923904 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:30.923875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxz26\" (UniqueName: \"kubernetes.io/projected/7e1af04f-1277-4565-8f0b-344bce74c765-kube-api-access-gxz26\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:31.090679 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.090638 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:31.254341 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.254297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:31.317081 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.317041 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e248df42-51d1-4bbc-9f34-a66e12596668" path="/var/lib/kubelet/pods/e248df42-51d1-4bbc-9f34-a66e12596668/volumes" Sep 15 20:59:31.319741 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.319707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:31.421214 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.421179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:31.423718 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.423691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:31.632944 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.632851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:31.664498 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.664450 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:31.946781 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.946744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q"] Sep 15 20:59:31.947817 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:59:31.947784 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1af04f_1277_4565_8f0b_344bce74c765.slice/crio-42f8edaeaac09021d1ba90b0160542bc560dcb78265e214cfb3f20fc3b42c683 WatchSource:0}: Error finding container 42f8edaeaac09021d1ba90b0160542bc560dcb78265e214cfb3f20fc3b42c683: Status 404 returned error can't find the container with id 42f8edaeaac09021d1ba90b0160542bc560dcb78265e214cfb3f20fc3b42c683 Sep 15 20:59:31.949318 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:31.949298 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 20:59:32.451099 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:32.451063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:32.662511 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:32.662394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" event={"ID":"7e1af04f-1277-4565-8f0b-344bce74c765","Type":"ContainerStarted","Data":"5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17"} Sep 15 20:59:32.662662 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:32.662586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:32.662662 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:32.662625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" event={"ID":"7e1af04f-1277-4565-8f0b-344bce74c765","Type":"ContainerStarted","Data":"42f8edaeaac09021d1ba90b0160542bc560dcb78265e214cfb3f20fc3b42c683"} Sep 15 20:59:32.681266 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:32.681200 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" podStartSLOduration=2.247834887 podStartE2EDuration="2.681179769s" totalImagesPullingTime="433.344882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:32.678663104 +0000 UTC m=+1361.808399687" watchObservedRunningTime="2026-09-15 20:59:32.681179769 +0000 UTC m=+1361.810916392" Sep 15 20:59:33.123695 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:33.123658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:33.810236 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:33.810201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:34.433542 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:34.433495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:35.077346 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:35.077305 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:36.055935 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:36.055893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:36.156897 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:36.156859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:36.423704 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:36.423592 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:36.523823 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:36.523786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:36.767922 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:36.767886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:37.099080 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:37.098991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:37.191599 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:37.191551 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:37.820273 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:37.820236 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:38.658681 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:38.658640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:38.675214 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:38.675185 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 20:59:38.758569 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:38.758530 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:38.907438 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:38.907366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:39.002459 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:39.002422 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:39.434603 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:39.434482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:41.884229 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:41.884190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:47.784123 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:47.784087 2580 generic.go:356] "Generic (PLEG): container finished" podID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerID="49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6" exitCode=0 Sep 15 20:59:47.784557 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:47.784129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" event={"ID":"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce","Type":"ContainerDied","Data":"49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6"} Sep 15 20:59:47.784603 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:47.784578 2580 scope.go:117] "RemoveContainer" containerID="49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6" Sep 15 20:59:47.785828 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:47.785794 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:47.785950 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:47.785880 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:47.785950 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:47.785929 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:48.793722 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:48.793686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" event={"ID":"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce","Type":"ContainerStarted","Data":"04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1"} Sep 15 20:59:49.578868 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:49.578830 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:49.579131 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:49.579101 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 15 20:59:49.579248 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:49.579176 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 15 20:59:50.579398 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:50.579355 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 15 20:59:50.579846 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:50.579442 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 15 20:59:51.578669 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:51.578624 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 15 20:59:51.578850 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:51.578696 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 15 20:59:51.720594 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:51.720500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:52.578583 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:52.578547 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 15 20:59:52.578785 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:52.578602 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 15 20:59:53.012302 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:53.012208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:53.578502 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:53.578460 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 15 20:59:53.578721 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:53.578528 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 15 20:59:53.653618 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:53.653577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:54.344784 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:54.344747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:54.582534 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:54.582505 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:54.582929 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:54.582903 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:54.583545 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:54.583523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 20:59:55.024513 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:55.024481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:55.841584 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:55.841550 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:56.460508 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:56.460478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:57.467519 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:57.467479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:57.778909 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:57.778872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:58.115717 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.115616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:58.424449 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.424347 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf"] Sep 15 20:59:58.429297 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.429278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.431535 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.431514 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3825fc-openshift-default-dockercfg-j2d2k\"" Sep 15 20:59:58.440313 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.440287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf"] Sep 15 20:59:58.446668 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.446642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:58.522583 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-certs\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-credential-socket\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljdf\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-kube-api-access-qljdf\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-socket\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-envoy\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b997461-188d-4750-b399-d93bc7a537ed-istio-podinfo\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-istio-token\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-data\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.523024 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.522993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b997461-188d-4750-b399-d93bc7a537ed-istiod-ca-cert\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624360 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-certs\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624543 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-credential-socket\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624543 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljdf\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-kube-api-access-qljdf\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624543 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-socket\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624543 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-envoy\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b997461-188d-4750-b399-d93bc7a537ed-istio-podinfo\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-istio-token\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624758 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-data\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b997461-188d-4750-b399-d93bc7a537ed-istiod-ca-cert\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-certs\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.624932 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.624871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-credential-socket\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.625086 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.625064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-data\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.625160 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.625138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-socket\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.625467 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.625450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b997461-188d-4750-b399-d93bc7a537ed-istiod-ca-cert\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.626921 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.626898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-envoy\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.627088 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.627068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b997461-188d-4750-b399-d93bc7a537ed-istio-podinfo\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.635936 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.635915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljdf\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-kube-api-access-qljdf\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.639496 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.639476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-istio-token\") pod \"e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:58.741061 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:58.740982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 20:59:59.021214 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.021167 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf"] Sep 15 20:59:59.022790 ip-10-0-139-201 kubenswrapper[2580]: W0915 20:59:59.022740 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b997461_188d_4750_b399_d93bc7a537ed.slice/crio-aac2e75dc503fda0a45a6a37c35b557f25e98e06576e7e98efd7f5b89d30199f WatchSource:0}: Error finding container aac2e75dc503fda0a45a6a37c35b557f25e98e06576e7e98efd7f5b89d30199f: Status 404 returned error can't find the container with id aac2e75dc503fda0a45a6a37c35b557f25e98e06576e7e98efd7f5b89d30199f Sep 15 20:59:59.025851 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.025813 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:59.025957 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.025915 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:59.026006 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.025961 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 20:59:59.176236 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.176202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:59.702102 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.702060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 20:59:59.869335 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.869300 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" event={"ID":"7b997461-188d-4750-b399-d93bc7a537ed","Type":"ContainerStarted","Data":"aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40"} Sep 15 20:59:59.869533 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.869353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" event={"ID":"7b997461-188d-4750-b399-d93bc7a537ed","Type":"ContainerStarted","Data":"aac2e75dc503fda0a45a6a37c35b557f25e98e06576e7e98efd7f5b89d30199f"} Sep 15 20:59:59.889712 ip-10-0-139-201 kubenswrapper[2580]: I0915 20:59:59.889665 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podStartSLOduration=1.889648909 podStartE2EDuration="1.889648909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:59.886291365 +0000 UTC m=+1389.016027948" watchObservedRunningTime="2026-09-15 20:59:59.889648909 +0000 UTC m=+1389.019385491" Sep 15 21:00:00.143568 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.143527 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv"] Sep 15 21:00:00.147565 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.147539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" Sep 15 21:00:00.150183 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.150159 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\"" Sep 15 21:00:00.157735 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.157656 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv"] Sep 15 21:00:00.162572 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.162545 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" Sep 15 21:00:00.448082 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.448051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv"] Sep 15 21:00:00.449199 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:00:00.449166 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbf6735_09ce_403a_a7c6_2f35572fb7eb.slice/crio-106138b2f7e70833f0fb9b4ebe58cdae81bab8c9a73403d087cd6ce58ae0bc91 WatchSource:0}: Error finding container 106138b2f7e70833f0fb9b4ebe58cdae81bab8c9a73403d087cd6ce58ae0bc91: Status 404 returned error can't find the container with id 106138b2f7e70833f0fb9b4ebe58cdae81bab8c9a73403d087cd6ce58ae0bc91 Sep 15 21:00:00.741553 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.741460 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:00.747033 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.747007 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:00.876592 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.876549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" event={"ID":"6bbf6735-09ce-403a-a7c6-2f35572fb7eb","Type":"ContainerStarted","Data":"106138b2f7e70833f0fb9b4ebe58cdae81bab8c9a73403d087cd6ce58ae0bc91"} Sep 15 21:00:00.876912 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.876895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:00.877968 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:00.877945 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:01.738076 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:01.738040 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:01.890867 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:01.890828 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:02.000542 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:02.000486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:02.665302 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:02.665244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:03.669775 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:03.669699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:03.902561 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:03.902520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" event={"ID":"6bbf6735-09ce-403a-a7c6-2f35572fb7eb","Type":"ContainerStarted","Data":"e2b74cc8d317513a415fb8b59c6f7dde12132d293b0432e8ea05716232733805"} Sep 15 21:00:03.918595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:03.918531 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" podStartSLOduration=1.225631402 podStartE2EDuration="3.918509293s" totalImagesPullingTime="2.692877891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:03.91462546 +0000 UTC m=+1393.044362044" watchObservedRunningTime="2026-09-15 21:00:03.918509293 +0000 UTC m=+1393.048245876" Sep 15 21:00:04.626159 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:04.626058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:04.911042 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:04.910938 2580 generic.go:356] "Generic (PLEG): container finished" podID="6bbf6735-09ce-403a-a7c6-2f35572fb7eb" containerID="e2b74cc8d317513a415fb8b59c6f7dde12132d293b0432e8ea05716232733805" exitCode=0 Sep 15 21:00:04.911042 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:04.911011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" event={"ID":"6bbf6735-09ce-403a-a7c6-2f35572fb7eb","Type":"ContainerDied","Data":"e2b74cc8d317513a415fb8b59c6f7dde12132d293b0432e8ea05716232733805"} Sep 15 21:00:05.038021 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.037977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:05.694854 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.694820 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j"] Sep 15 21:00:05.698999 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.698974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.700921 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.700897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-3825fc-serving-cert\"" Sep 15 21:00:05.708883 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.708860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j"] Sep 15 21:00:05.732313 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.732271 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:05.811867 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.811796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793f89d-e418-41d8-94ec-3d7b34ed15b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.812098 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.811994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.812098 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.812046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4brs2\" (UniqueName: \"kubernetes.io/projected/0793f89d-e418-41d8-94ec-3d7b34ed15b8-kube-api-access-4brs2\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.812218 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.812104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.913885 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.913852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.914359 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.913899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4brs2\" (UniqueName: \"kubernetes.io/projected/0793f89d-e418-41d8-94ec-3d7b34ed15b8-kube-api-access-4brs2\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.914359 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.913949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.914359 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.914003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793f89d-e418-41d8-94ec-3d7b34ed15b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.914359 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:05.914041 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-serving-cert: secret "maas-api-e2e-shared-b-w2-3825fc-serving-cert" not found Sep 15 21:00:05.914359 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:05.914142 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls podName:0793f89d-e418-41d8-94ec-3d7b34ed15b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:00:06.414119155 +0000 UTC m=+1395.543855730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" (UID: "0793f89d-e418-41d8-94ec-3d7b34ed15b8") : secret "maas-api-e2e-shared-b-w2-3825fc-serving-cert" not found Sep 15 21:00:05.914847 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.914819 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793f89d-e418-41d8-94ec-3d7b34ed15b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.916843 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.916813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:05.922984 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:05.922959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4brs2\" (UniqueName: \"kubernetes.io/projected/0793f89d-e418-41d8-94ec-3d7b34ed15b8-kube-api-access-4brs2\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:06.066799 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.066772 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" Sep 15 21:00:06.189181 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.189143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:06.274536 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.274500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:06.357588 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.357554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:06.421488 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.421450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:06.423841 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.423815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:06.613572 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.613484 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:06.902961 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.902934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j"] Sep 15 21:00:06.905071 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:00:06.905043 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0793f89d_e418_41d8_94ec_3d7b34ed15b8.slice/crio-38742bf2dfc001271a6b3f768534053dd1b3d50c84324ef05b641bc1142bdb97 WatchSource:0}: Error finding container 38742bf2dfc001271a6b3f768534053dd1b3d50c84324ef05b641bc1142bdb97: Status 404 returned error can't find the container with id 38742bf2dfc001271a6b3f768534053dd1b3d50c84324ef05b641bc1142bdb97 Sep 15 21:00:06.929034 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.929003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" event={"ID":"0793f89d-e418-41d8-94ec-3d7b34ed15b8","Type":"ContainerStarted","Data":"38742bf2dfc001271a6b3f768534053dd1b3d50c84324ef05b641bc1142bdb97"} Sep 15 21:00:06.930545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.930524 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" Sep 15 21:00:06.930545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.930535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv" event={"ID":"6bbf6735-09ce-403a-a7c6-2f35572fb7eb","Type":"ContainerDied","Data":"106138b2f7e70833f0fb9b4ebe58cdae81bab8c9a73403d087cd6ce58ae0bc91"} Sep 15 21:00:06.930714 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:06.930575 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106138b2f7e70833f0fb9b4ebe58cdae81bab8c9a73403d087cd6ce58ae0bc91" Sep 15 21:00:07.940202 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:07.940160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" event={"ID":"0793f89d-e418-41d8-94ec-3d7b34ed15b8","Type":"ContainerStarted","Data":"1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3"} Sep 15 21:00:07.940640 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:07.940365 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:07.955756 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:07.955691 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" podStartSLOduration=2.503669023 podStartE2EDuration="2.955660553s" totalImagesPullingTime="451.99153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:07.955274692 +0000 UTC m=+1397.085011275" watchObservedRunningTime="2026-09-15 21:00:07.955660553 +0000 UTC m=+1397.085397136" Sep 15 21:00:13.952595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:13.952559 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:00:23.808996 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:23.808955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:23.910519 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:23.910479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:28.075770 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:28.075733 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:28.207046 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:28.207000 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:28.419233 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:28.419132 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:28.526311 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:28.526232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:28.627101 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:28.627063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:29.090938 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:29.090837 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b997461-188d-4750-b399-d93bc7a537ed" containerID="aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40" exitCode=0 Sep 15 21:00:29.090938 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:29.090919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" event={"ID":"7b997461-188d-4750-b399-d93bc7a537ed","Type":"ContainerDied","Data":"aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40"} Sep 15 21:00:29.091419 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:29.091375 2580 scope.go:117] "RemoveContainer" containerID="aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40" Sep 15 21:00:29.092545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:29.092514 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:00:29.092650 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:29.092579 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:00:29.092650 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:29.092618 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:00:29.340192 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:29.340149 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:30.014155 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.014076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:30.100169 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.100136 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" event={"ID":"7b997461-188d-4750-b399-d93bc7a537ed","Type":"ContainerStarted","Data":"bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98"} Sep 15 21:00:30.132244 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.132153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:30.704459 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.704420 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:30.741671 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.741631 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:30.742865 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.742841 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:00:30.742990 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.742893 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 15 21:00:30.877310 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:30.877268 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:31.348351 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:31.348314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:31.741662 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:31.741578 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:00:31.741662 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:31.741636 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 15 21:00:31.955158 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:31.955123 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:32.055352 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.055309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:32.694160 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.694123 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7752-43a53b"] Sep 15 21:00:32.694697 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.694682 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbf6735-09ce-403a-a7c6-2f35572fb7eb" containerName="cleanup" Sep 15 21:00:32.694750 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.694698 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbf6735-09ce-403a-a7c6-2f35572fb7eb" containerName="cleanup" Sep 15 21:00:32.694816 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.694806 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbf6735-09ce-403a-a7c6-2f35572fb7eb" containerName="cleanup" Sep 15 21:00:32.698482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.698455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:32.700923 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.700897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cs2t2\"" Sep 15 21:00:32.701045 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.700940 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-43a53b"] Sep 15 21:00:32.742429 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.741870 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:00:32.742429 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.741945 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 15 21:00:32.827426 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.815233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnhk\" (UniqueName: \"kubernetes.io/projected/6fdad343-c758-4950-8c0f-9c0c82c4336d-kube-api-access-2pnhk\") pod \"test-curl-7752-43a53b\" (UID: \"6fdad343-c758-4950-8c0f-9c0c82c4336d\") " pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:32.827641 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.827461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:32.917103 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.917063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnhk\" (UniqueName: \"kubernetes.io/projected/6fdad343-c758-4950-8c0f-9c0c82c4336d-kube-api-access-2pnhk\") pod \"test-curl-7752-43a53b\" (UID: \"6fdad343-c758-4950-8c0f-9c0c82c4336d\") " pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:32.941460 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:32.939468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnhk\" (UniqueName: \"kubernetes.io/projected/6fdad343-c758-4950-8c0f-9c0c82c4336d-kube-api-access-2pnhk\") pod \"test-curl-7752-43a53b\" (UID: \"6fdad343-c758-4950-8c0f-9c0c82c4336d\") " pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:33.009536 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:33.009491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:33.298342 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:33.298313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-43a53b"] Sep 15 21:00:33.299115 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:00:33.299092 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fdad343_c758_4950_8c0f_9c0c82c4336d.slice/crio-23c438f718ff0275a169885fe7a044871441b3687b24237a9290a0d8f1ad1958 WatchSource:0}: Error finding container 23c438f718ff0275a169885fe7a044871441b3687b24237a9290a0d8f1ad1958: Status 404 returned error can't find the container with id 23c438f718ff0275a169885fe7a044871441b3687b24237a9290a0d8f1ad1958 Sep 15 21:00:33.627546 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:33.627440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:33.742069 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:33.742029 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:00:33.742477 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:33.742082 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 15 21:00:34.154025 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.153986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-43a53b" event={"ID":"6fdad343-c758-4950-8c0f-9c0c82c4336d","Type":"ContainerStarted","Data":"66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501"} Sep 15 21:00:34.154217 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.154034 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-43a53b" event={"ID":"6fdad343-c758-4950-8c0f-9c0c82c4336d","Type":"ContainerStarted","Data":"23c438f718ff0275a169885fe7a044871441b3687b24237a9290a0d8f1ad1958"} Sep 15 21:00:34.168564 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.168506 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7752-43a53b" podStartSLOduration=2.168491559 podStartE2EDuration="2.168491559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:34.167149096 +0000 UTC m=+1423.296885679" watchObservedRunningTime="2026-09-15 21:00:34.168491559 +0000 UTC m=+1423.298228142" Sep 15 21:00:34.742113 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.742072 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 15 21:00:34.742501 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.742143 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 15 21:00:34.795656 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.795620 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7752-43a53b"] Sep 15 21:00:34.798502 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.798474 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7752-43a53b"] Sep 15 21:00:34.938547 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.938458 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7752-bce29c"] Sep 15 21:00:34.939548 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.939511 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fdad343-c758-4950-8c0f-9c0c82c4336d" containerName="test-curl-7752-43a53b" Sep 15 21:00:34.939711 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.939697 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fdad343-c758-4950-8c0f-9c0c82c4336d" containerName="test-curl-7752-43a53b" Sep 15 21:00:34.939935 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.939919 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fdad343-c758-4950-8c0f-9c0c82c4336d" containerName="test-curl-7752-43a53b" Sep 15 21:00:34.943912 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.943887 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-bce29c"] Sep 15 21:00:34.944017 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:34.943986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:35.056784 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.056749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gbq\" (UniqueName: \"kubernetes.io/projected/7d158322-d1ef-407c-a471-00634152eec4-kube-api-access-p2gbq\") pod \"test-curl-7752-bce29c\" (UID: \"7d158322-d1ef-407c-a471-00634152eec4\") " pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:35.157549 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.157510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gbq\" (UniqueName: \"kubernetes.io/projected/7d158322-d1ef-407c-a471-00634152eec4-kube-api-access-p2gbq\") pod \"test-curl-7752-bce29c\" (UID: \"7d158322-d1ef-407c-a471-00634152eec4\") " pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:35.162441 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:35.162399 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7752-43a53b" container="test-curl-7752-43a53b" Sep 15 21:00:35.165994 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.165962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gbq\" (UniqueName: \"kubernetes.io/projected/7d158322-d1ef-407c-a471-00634152eec4-kube-api-access-p2gbq\") pod \"test-curl-7752-bce29c\" (UID: \"7d158322-d1ef-407c-a471-00634152eec4\") " pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:35.256516 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.256485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:35.530095 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.530061 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-bce29c"] Sep 15 21:00:35.531557 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:00:35.531530 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d158322_d1ef_407c_a471_00634152eec4.slice/crio-95417706ebdd6c784a1b788a49b4d313c8e196bf223d6b62e272a2c239e6dbee WatchSource:0}: Error finding container 95417706ebdd6c784a1b788a49b4d313c8e196bf223d6b62e272a2c239e6dbee: Status 404 returned error can't find the container with id 95417706ebdd6c784a1b788a49b4d313c8e196bf223d6b62e272a2c239e6dbee Sep 15 21:00:35.745695 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.745664 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:35.746766 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.746742 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:00:35.993275 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:35.993191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:36.170063 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.170028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-bce29c" event={"ID":"7d158322-d1ef-407c-a471-00634152eec4","Type":"ContainerStarted","Data":"3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656"} Sep 15 21:00:36.170246 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.170073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-bce29c" event={"ID":"7d158322-d1ef-407c-a471-00634152eec4","Type":"ContainerStarted","Data":"95417706ebdd6c784a1b788a49b4d313c8e196bf223d6b62e272a2c239e6dbee"} Sep 15 21:00:36.170488 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.170452 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7752-43a53b" podUID="6fdad343-c758-4950-8c0f-9c0c82c4336d" containerName="test-curl-7752-43a53b" containerID="cri-o://66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501" gracePeriod=2 Sep 15 21:00:36.184712 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.184649 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7752-bce29c" podStartSLOduration=2.184630727 podStartE2EDuration="2.184630727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:36.184079326 +0000 UTC m=+1425.313815908" watchObservedRunningTime="2026-09-15 21:00:36.184630727 +0000 UTC m=+1425.314367309" Sep 15 21:00:36.751267 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.751237 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7752-bce29c"] Sep 15 21:00:36.755905 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.755877 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7752-bce29c"] Sep 15 21:00:36.868114 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.868081 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7752-a4663e"] Sep 15 21:00:36.868578 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.868565 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d158322-d1ef-407c-a471-00634152eec4" containerName="test-curl-7752-bce29c" Sep 15 21:00:36.868639 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.868580 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d158322-d1ef-407c-a471-00634152eec4" containerName="test-curl-7752-bce29c" Sep 15 21:00:36.868678 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.868667 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d158322-d1ef-407c-a471-00634152eec4" containerName="test-curl-7752-bce29c" Sep 15 21:00:36.872031 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.872016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:36.877792 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.877769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-a4663e"] Sep 15 21:00:36.980295 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:36.980256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkxb\" (UniqueName: \"kubernetes.io/projected/eeda2b2b-ecdd-4e9f-944a-352fe25e277a-kube-api-access-vvkxb\") pod \"test-curl-7752-a4663e\" (UID: \"eeda2b2b-ecdd-4e9f-944a-352fe25e277a\") " pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:37.081084 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:37.080999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkxb\" (UniqueName: \"kubernetes.io/projected/eeda2b2b-ecdd-4e9f-944a-352fe25e277a-kube-api-access-vvkxb\") pod \"test-curl-7752-a4663e\" (UID: \"eeda2b2b-ecdd-4e9f-944a-352fe25e277a\") " pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:37.089504 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:37.089478 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkxb\" (UniqueName: \"kubernetes.io/projected/eeda2b2b-ecdd-4e9f-944a-352fe25e277a-kube-api-access-vvkxb\") pod \"test-curl-7752-a4663e\" (UID: \"eeda2b2b-ecdd-4e9f-944a-352fe25e277a\") " pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:37.176486 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:37.176454 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7752-bce29c" container="test-curl-7752-bce29c" Sep 15 21:00:37.182340 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:37.182322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:37.455308 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:37.455277 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-a4663e"] Sep 15 21:00:37.458267 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:00:37.458239 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeda2b2b_ecdd_4e9f_944a_352fe25e277a.slice/crio-fa97bfe83b9dc4d79de8ec34989470592442dc6e246a701196d4b756ce31bd70 WatchSource:0}: Error finding container fa97bfe83b9dc4d79de8ec34989470592442dc6e246a701196d4b756ce31bd70: Status 404 returned error can't find the container with id fa97bfe83b9dc4d79de8ec34989470592442dc6e246a701196d4b756ce31bd70 Sep 15 21:00:38.183923 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.183887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-a4663e" event={"ID":"eeda2b2b-ecdd-4e9f-944a-352fe25e277a","Type":"ContainerStarted","Data":"2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f"} Sep 15 21:00:38.184285 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.183939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-a4663e" event={"ID":"eeda2b2b-ecdd-4e9f-944a-352fe25e277a","Type":"ContainerStarted","Data":"fa97bfe83b9dc4d79de8ec34989470592442dc6e246a701196d4b756ce31bd70"} Sep 15 21:00:38.184285 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.184054 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7752-bce29c" podUID="7d158322-d1ef-407c-a471-00634152eec4" containerName="test-curl-7752-bce29c" containerID="cri-o://3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656" gracePeriod=2 Sep 15 21:00:38.198160 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.198118 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7752-a4663e" podStartSLOduration=2.19809936 podStartE2EDuration="2.19809936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:38.196770593 +0000 UTC m=+1427.326507176" watchObservedRunningTime="2026-09-15 21:00:38.19809936 +0000 UTC m=+1427.327835944" Sep 15 21:00:38.373504 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.373470 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:38.496782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.496686 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pnhk\" (UniqueName: \"kubernetes.io/projected/6fdad343-c758-4950-8c0f-9c0c82c4336d-kube-api-access-2pnhk\") pod \"6fdad343-c758-4950-8c0f-9c0c82c4336d\" (UID: \"6fdad343-c758-4950-8c0f-9c0c82c4336d\") " Sep 15 21:00:38.498884 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.498850 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdad343-c758-4950-8c0f-9c0c82c4336d-kube-api-access-2pnhk" (OuterVolumeSpecName: "kube-api-access-2pnhk") pod "6fdad343-c758-4950-8c0f-9c0c82c4336d" (UID: "6fdad343-c758-4950-8c0f-9c0c82c4336d"). InnerVolumeSpecName "kube-api-access-2pnhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:00:38.604847 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.599527 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pnhk\" (UniqueName: \"kubernetes.io/projected/6fdad343-c758-4950-8c0f-9c0c82c4336d-kube-api-access-2pnhk\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:00:38.790956 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.790915 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7752-a4663e"] Sep 15 21:00:38.795876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.795845 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7752-a4663e"] Sep 15 21:00:38.909474 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.909437 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7752-27cff6"] Sep 15 21:00:38.910044 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.910024 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeda2b2b-ecdd-4e9f-944a-352fe25e277a" containerName="test-curl-7752-a4663e" Sep 15 21:00:38.910044 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.910046 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeda2b2b-ecdd-4e9f-944a-352fe25e277a" containerName="test-curl-7752-a4663e" Sep 15 21:00:38.910207 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.910183 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeda2b2b-ecdd-4e9f-944a-352fe25e277a" containerName="test-curl-7752-a4663e" Sep 15 21:00:38.913876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.913854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:38.919958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:38.919931 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-27cff6"] Sep 15 21:00:39.003211 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.003177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtwj\" (UniqueName: \"kubernetes.io/projected/1486413c-178a-43b5-ba27-a39850380cad-kube-api-access-lbtwj\") pod \"test-curl-7752-27cff6\" (UID: \"1486413c-178a-43b5-ba27-a39850380cad\") " pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:39.104105 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.104028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtwj\" (UniqueName: \"kubernetes.io/projected/1486413c-178a-43b5-ba27-a39850380cad-kube-api-access-lbtwj\") pod \"test-curl-7752-27cff6\" (UID: \"1486413c-178a-43b5-ba27-a39850380cad\") " pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:39.114263 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.114232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtwj\" (UniqueName: \"kubernetes.io/projected/1486413c-178a-43b5-ba27-a39850380cad-kube-api-access-lbtwj\") pod \"test-curl-7752-27cff6\" (UID: \"1486413c-178a-43b5-ba27-a39850380cad\") " pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:39.191694 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.191662 2580 generic.go:356] "Generic (PLEG): container finished" podID="6fdad343-c758-4950-8c0f-9c0c82c4336d" containerID="66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501" exitCode=137 Sep 15 21:00:39.192133 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.191729 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:39.192133 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.191761 2580 scope.go:117] "RemoveContainer" containerID="66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501" Sep 15 21:00:39.192133 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:39.192033 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7752-a4663e" container="test-curl-7752-a4663e" Sep 15 21:00:39.201821 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.201799 2580 scope.go:117] "RemoveContainer" containerID="66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501" Sep 15 21:00:39.202070 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:39.202053 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501\": container with ID starting with 66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501 not found: ID does not exist" containerID="66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501" Sep 15 21:00:39.202122 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.202078 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501"} err="failed to get container status \"66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501\": rpc error: code = NotFound desc = could not find container \"66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501\": container with ID starting with 66ed3984798e7ffa83b3071b4c63e9a2a39e15e5b8e91ea8bd02ea3741efa501 not found: ID does not exist" Sep 15 21:00:39.202777 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:39.202750 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7752-43a53b\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="6fdad343-c758-4950-8c0f-9c0c82c4336d" pod="opendatahub/test-curl-7752-43a53b" Sep 15 21:00:39.225789 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.225768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:39.315992 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.315958 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fdad343-c758-4950-8c0f-9c0c82c4336d" path="/var/lib/kubelet/pods/6fdad343-c758-4950-8c0f-9c0c82c4336d/volumes" Sep 15 21:00:39.499595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.499563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-27cff6"] Sep 15 21:00:39.500716 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:00:39.500688 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1486413c_178a_43b5_ba27_a39850380cad.slice/crio-90a11e1baff9db6f7cb0adcda51d09d7f8eb6a881d1785be827fbe2174c29281 WatchSource:0}: Error finding container 90a11e1baff9db6f7cb0adcda51d09d7f8eb6a881d1785be827fbe2174c29281: Status 404 returned error can't find the container with id 90a11e1baff9db6f7cb0adcda51d09d7f8eb6a881d1785be827fbe2174c29281 Sep 15 21:00:39.834283 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:39.834245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:40.201543 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.201510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-27cff6" event={"ID":"1486413c-178a-43b5-ba27-a39850380cad","Type":"ContainerStarted","Data":"c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd"} Sep 15 21:00:40.201862 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.201562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-27cff6" event={"ID":"1486413c-178a-43b5-ba27-a39850380cad","Type":"ContainerStarted","Data":"90a11e1baff9db6f7cb0adcda51d09d7f8eb6a881d1785be827fbe2174c29281"} Sep 15 21:00:40.201862 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.201576 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7752-a4663e" podUID="eeda2b2b-ecdd-4e9f-944a-352fe25e277a" containerName="test-curl-7752-a4663e" containerID="cri-o://2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f" gracePeriod=2 Sep 15 21:00:40.215146 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.215092 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7752-27cff6" podStartSLOduration=2.215070404 podStartE2EDuration="2.215070404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:40.214589235 +0000 UTC m=+1429.344325820" watchObservedRunningTime="2026-09-15 21:00:40.215070404 +0000 UTC m=+1429.344806987" Sep 15 21:00:40.377043 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.377018 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:40.519686 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.519652 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2gbq\" (UniqueName: \"kubernetes.io/projected/7d158322-d1ef-407c-a471-00634152eec4-kube-api-access-p2gbq\") pod \"7d158322-d1ef-407c-a471-00634152eec4\" (UID: \"7d158322-d1ef-407c-a471-00634152eec4\") " Sep 15 21:00:40.521766 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.521732 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d158322-d1ef-407c-a471-00634152eec4-kube-api-access-p2gbq" (OuterVolumeSpecName: "kube-api-access-p2gbq") pod "7d158322-d1ef-407c-a471-00634152eec4" (UID: "7d158322-d1ef-407c-a471-00634152eec4"). InnerVolumeSpecName "kube-api-access-p2gbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:00:40.621468 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.621430 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gbq\" (UniqueName: \"kubernetes.io/projected/7d158322-d1ef-407c-a471-00634152eec4-kube-api-access-p2gbq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:00:40.805163 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.805076 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7752-27cff6"] Sep 15 21:00:40.809982 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:40.809947 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7752-27cff6"] Sep 15 21:00:41.209043 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.208945 2580 generic.go:356] "Generic (PLEG): container finished" podID="7d158322-d1ef-407c-a471-00634152eec4" containerID="3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656" exitCode=137 Sep 15 21:00:41.209043 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.208990 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:41.209043 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.209039 2580 scope.go:117] "RemoveContainer" containerID="3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656" Sep 15 21:00:41.209643 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:41.209439 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7752-27cff6" container="test-curl-7752-27cff6" Sep 15 21:00:41.221754 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.221734 2580 scope.go:117] "RemoveContainer" containerID="3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656" Sep 15 21:00:41.222037 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:41.222018 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656\": container with ID starting with 3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656 not found: ID does not exist" containerID="3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656" Sep 15 21:00:41.222103 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.222051 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656"} err="failed to get container status \"3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656\": rpc error: code = NotFound desc = could not find container \"3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656\": container with ID starting with 3dc15adafbc4ab3d5122996878f4a57809848845183d6551b99128f92ed9b656 not found: ID does not exist" Sep 15 21:00:41.222818 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:41.222794 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7752-bce29c\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="7d158322-d1ef-407c-a471-00634152eec4" pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:41.313619 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:41.313560 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7752-bce29c\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="7d158322-d1ef-407c-a471-00634152eec4" pod="opendatahub/test-curl-7752-bce29c" Sep 15 21:00:41.314951 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.314920 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d158322-d1ef-407c-a471-00634152eec4" path="/var/lib/kubelet/pods/7d158322-d1ef-407c-a471-00634152eec4/volumes" Sep 15 21:00:41.386324 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.386279 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:41.687125 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:41.687065 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:42.218501 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:42.218398 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7752-27cff6" podUID="1486413c-178a-43b5-ba27-a39850380cad" containerName="test-curl-7752-27cff6" containerID="cri-o://c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd" gracePeriod=2 Sep 15 21:00:42.406810 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:42.406779 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:42.526009 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:42.525977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:42.542220 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:42.542194 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvkxb\" (UniqueName: \"kubernetes.io/projected/eeda2b2b-ecdd-4e9f-944a-352fe25e277a-kube-api-access-vvkxb\") pod \"eeda2b2b-ecdd-4e9f-944a-352fe25e277a\" (UID: \"eeda2b2b-ecdd-4e9f-944a-352fe25e277a\") " Sep 15 21:00:42.544272 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:42.544250 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeda2b2b-ecdd-4e9f-944a-352fe25e277a-kube-api-access-vvkxb" (OuterVolumeSpecName: "kube-api-access-vvkxb") pod "eeda2b2b-ecdd-4e9f-944a-352fe25e277a" (UID: "eeda2b2b-ecdd-4e9f-944a-352fe25e277a"). InnerVolumeSpecName "kube-api-access-vvkxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:00:42.643044 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:42.643015 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvkxb\" (UniqueName: \"kubernetes.io/projected/eeda2b2b-ecdd-4e9f-944a-352fe25e277a-kube-api-access-vvkxb\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:00:43.230052 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:43.230009 2580 generic.go:356] "Generic (PLEG): container finished" podID="eeda2b2b-ecdd-4e9f-944a-352fe25e277a" containerID="2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f" exitCode=137 Sep 15 21:00:43.230485 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:43.230062 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:43.230485 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:43.230100 2580 scope.go:117] "RemoveContainer" containerID="2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f" Sep 15 21:00:43.239692 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:43.239674 2580 scope.go:117] "RemoveContainer" containerID="2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f" Sep 15 21:00:43.239962 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:43.239943 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f\": container with ID starting with 2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f not found: ID does not exist" containerID="2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f" Sep 15 21:00:43.240008 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:43.239969 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f"} err="failed to get container status \"2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f\": rpc error: code = NotFound desc = could not find container \"2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f\": container with ID starting with 2a2c0479747e546c664c82b24ab31950b2d04a4f50511b334d3cc44571880e8f not found: ID does not exist" Sep 15 21:00:43.240306 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:43.240288 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7752-a4663e\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="eeda2b2b-ecdd-4e9f-944a-352fe25e277a" pod="opendatahub/test-curl-7752-a4663e" Sep 15 21:00:43.322337 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:43.322294 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeda2b2b-ecdd-4e9f-944a-352fe25e277a" path="/var/lib/kubelet/pods/eeda2b2b-ecdd-4e9f-944a-352fe25e277a/volumes" Sep 15 21:00:44.411160 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:44.411132 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:44.460577 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:44.460542 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbtwj\" (UniqueName: \"kubernetes.io/projected/1486413c-178a-43b5-ba27-a39850380cad-kube-api-access-lbtwj\") pod \"1486413c-178a-43b5-ba27-a39850380cad\" (UID: \"1486413c-178a-43b5-ba27-a39850380cad\") " Sep 15 21:00:44.462538 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:44.462513 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1486413c-178a-43b5-ba27-a39850380cad-kube-api-access-lbtwj" (OuterVolumeSpecName: "kube-api-access-lbtwj") pod "1486413c-178a-43b5-ba27-a39850380cad" (UID: "1486413c-178a-43b5-ba27-a39850380cad"). InnerVolumeSpecName "kube-api-access-lbtwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:00:44.561737 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:44.561702 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtwj\" (UniqueName: \"kubernetes.io/projected/1486413c-178a-43b5-ba27-a39850380cad-kube-api-access-lbtwj\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:00:45.246194 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.246154 2580 generic.go:356] "Generic (PLEG): container finished" podID="1486413c-178a-43b5-ba27-a39850380cad" containerID="c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd" exitCode=137 Sep 15 21:00:45.246194 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.246196 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:45.246490 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.246248 2580 scope.go:117] "RemoveContainer" containerID="c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd" Sep 15 21:00:45.256323 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:45.256286 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7752-27cff6\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="1486413c-178a-43b5-ba27-a39850380cad" pod="opendatahub/test-curl-7752-27cff6" Sep 15 21:00:45.256570 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.256484 2580 scope.go:117] "RemoveContainer" containerID="c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd" Sep 15 21:00:45.256812 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:00:45.256785 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd\": container with ID starting with c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd not found: ID does not exist" containerID="c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd" Sep 15 21:00:45.256865 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.256812 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd"} err="failed to get container status \"c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd\": rpc error: code = NotFound desc = could not find container \"c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd\": container with ID starting with c2727db1001816e25a616b0724f3f5bc071f007278603648cf4971077c0dbacd not found: ID does not exist" Sep 15 21:00:45.315452 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.315391 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1486413c-178a-43b5-ba27-a39850380cad" path="/var/lib/kubelet/pods/1486413c-178a-43b5-ba27-a39850380cad/volumes" Sep 15 21:00:45.782569 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.782537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:45.842496 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:45.842460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:46.030312 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:46.030275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:46.937696 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:46.937659 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:47.616935 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:47.616899 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:50.015673 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:50.015628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:00:50.119663 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:00:50.119624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:00.273956 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:00.273923 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:01.263700 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:01.263664 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:01.961944 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:01.961912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:02.624507 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:02.624473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:03.420826 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:03.420787 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:04.119508 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:04.119471 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:04.652245 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:04.652206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:04.886390 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:04.886358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:05.835871 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:05.835832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:06.528782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:06.528739 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:07.245677 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:07.245624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:08.169465 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:08.169429 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:08.928514 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:08.928478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:09.667316 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:09.667281 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:10.444094 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:10.444057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:10.659342 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:10.659309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:11.307671 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:11.307633 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:12.125510 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:12.125473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:12.775526 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:12.775464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:12.875176 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:12.875143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:12.960738 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:12.960701 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:13.014015 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:13.013981 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:13.330716 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:13.330672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:13.431413 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:13.431367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:14.061764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:14.061732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:14.836918 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:14.836878 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:15.468876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:15.468844 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:16.236566 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:16.236529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:17.032467 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:17.032434 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:17.661175 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:17.661141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:18.703519 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:18.703484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:18.764195 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:18.764138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:19.144616 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:19.144573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:19.316568 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:19.316515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:19.392106 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:19.392062 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk"] Sep 15 21:01:19.392386 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:19.392331 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" containerID="cri-o://5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82" gracePeriod=30 Sep 15 21:01:19.417070 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:19.416995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:21.816633 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:21.816598 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:21.917063 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:21.917025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:22.238659 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:22.238578 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:22.800028 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:22.799983 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:22.999614 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:22.999586 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:24.664099 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.664035 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 21:01:24.684495 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684470 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-credential-socket\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.684638 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-envoy\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.684638 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684554 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb07e50-b590-4a0c-8244-c45d83329b75-istio-podinfo\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.684638 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684603 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb07e50-b590-4a0c-8244-c45d83329b75-istiod-ca-cert\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.684638 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684638 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-data\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.684869 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684731 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:24.684933 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684888 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-data" (OuterVolumeSpecName: "istio-data") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:24.684991 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684931 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-socket\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.684991 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684980 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmw8r\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-kube-api-access-jmw8r\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.685104 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.684997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb07e50-b590-4a0c-8244-c45d83329b75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:01:24.685104 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685038 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-istio-token\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.685242 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685110 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-certs\") pod \"edb07e50-b590-4a0c-8244-c45d83329b75\" (UID: \"edb07e50-b590-4a0c-8244-c45d83329b75\") " Sep 15 21:01:24.685525 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685442 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:24.685812 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685630 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:24.685812 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685715 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edb07e50-b590-4a0c-8244-c45d83329b75-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.685812 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685735 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.685812 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685747 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.685812 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.685759 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.687676 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.687641 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-kube-api-access-jmw8r" (OuterVolumeSpecName: "kube-api-access-jmw8r") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "kube-api-access-jmw8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:24.688273 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.688217 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-istio-token" (OuterVolumeSpecName: "istio-token") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:24.688612 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.688582 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edb07e50-b590-4a0c-8244-c45d83329b75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:01:24.688753 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.688730 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "edb07e50-b590-4a0c-8244-c45d83329b75" (UID: "edb07e50-b590-4a0c-8244-c45d83329b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:01:24.786827 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.786791 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.786827 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.786821 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edb07e50-b590-4a0c-8244-c45d83329b75-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.786827 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.786830 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmw8r\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-kube-api-access-jmw8r\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.787079 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.786840 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edb07e50-b590-4a0c-8244-c45d83329b75-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:24.787079 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:24.786849 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edb07e50-b590-4a0c-8244-c45d83329b75-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:25.483942 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.483904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:25.516171 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.516128 2580 generic.go:356] "Generic (PLEG): container finished" podID="edb07e50-b590-4a0c-8244-c45d83329b75" containerID="5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82" exitCode=0 Sep 15 21:01:25.516355 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.516177 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" Sep 15 21:01:25.516355 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.516217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" event={"ID":"edb07e50-b590-4a0c-8244-c45d83329b75","Type":"ContainerDied","Data":"5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82"} Sep 15 21:01:25.516355 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.516279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk" event={"ID":"edb07e50-b590-4a0c-8244-c45d83329b75","Type":"ContainerDied","Data":"c59a38028277b89cd50e51c7fa438428501539de2e2615f716b113b666aec9b8"} Sep 15 21:01:25.516355 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.516309 2580 scope.go:117] "RemoveContainer" containerID="5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82" Sep 15 21:01:25.527103 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.527085 2580 scope.go:117] "RemoveContainer" containerID="96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604" Sep 15 21:01:25.536947 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.536923 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk"] Sep 15 21:01:25.538825 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.538810 2580 scope.go:117] "RemoveContainer" containerID="5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82" Sep 15 21:01:25.539115 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:01:25.539095 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82\": container with ID starting with 5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82 not found: ID does not exist" containerID="5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82" Sep 15 21:01:25.539221 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.539127 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82"} err="failed to get container status \"5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82\": rpc error: code = NotFound desc = could not find container \"5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82\": container with ID starting with 5f07e6c760d0b89a24a4675698918638defc580f8d68f4b80097ba62ff0e2e82 not found: ID does not exist" Sep 15 21:01:25.539221 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.539151 2580 scope.go:117] "RemoveContainer" containerID="96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604" Sep 15 21:01:25.539516 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:01:25.539487 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604\": container with ID starting with 96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604 not found: ID does not exist" containerID="96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604" Sep 15 21:01:25.539619 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.539524 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604"} err="failed to get container status \"96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604\": rpc error: code = NotFound desc = could not find container \"96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604\": container with ID starting with 96e26f78b04ff874178d04ee67411fb5f6ed30aa26324ac0b89fee58e91f8604 not found: ID does not exist" Sep 15 21:01:25.540867 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.540849 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d5b5b9-openshift-default-5bd68b7b56-546rk"] Sep 15 21:01:25.580062 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:25.579985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:27.323765 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:27.323724 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" path="/var/lib/kubelet/pods/edb07e50-b590-4a0c-8244-c45d83329b75/volumes" Sep 15 21:01:30.218154 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.218116 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8"] Sep 15 21:01:30.218598 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.218388 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" podUID="00f05f58-7565-43f5-916e-1b4476f14add" containerName="maas-api" containerID="cri-o://85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492" gracePeriod=30 Sep 15 21:01:30.302822 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.302784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:30.367963 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.367906 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:30.468210 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.468100 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:30.497962 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.497937 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 21:01:30.552064 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.552033 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls\") pod \"00f05f58-7565-43f5-916e-1b4476f14add\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " Sep 15 21:01:30.552233 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.552164 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-metrics-certs\") pod \"00f05f58-7565-43f5-916e-1b4476f14add\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " Sep 15 21:01:30.552233 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.552192 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f05f58-7565-43f5-916e-1b4476f14add-openshift-service-ca\") pod \"00f05f58-7565-43f5-916e-1b4476f14add\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " Sep 15 21:01:30.552380 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.552244 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lb2q\" (UniqueName: \"kubernetes.io/projected/00f05f58-7565-43f5-916e-1b4476f14add-kube-api-access-4lb2q\") pod \"00f05f58-7565-43f5-916e-1b4476f14add\" (UID: \"00f05f58-7565-43f5-916e-1b4476f14add\") " Sep 15 21:01:30.552680 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.552648 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f05f58-7565-43f5-916e-1b4476f14add-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00f05f58-7565-43f5-916e-1b4476f14add" (UID: "00f05f58-7565-43f5-916e-1b4476f14add"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:01:30.553903 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.553865 2580 generic.go:356] "Generic (PLEG): container finished" podID="00f05f58-7565-43f5-916e-1b4476f14add" containerID="85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492" exitCode=0 Sep 15 21:01:30.554015 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.553929 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" Sep 15 21:01:30.554015 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.553954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" event={"ID":"00f05f58-7565-43f5-916e-1b4476f14add","Type":"ContainerDied","Data":"85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492"} Sep 15 21:01:30.554178 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.554030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8" event={"ID":"00f05f58-7565-43f5-916e-1b4476f14add","Type":"ContainerDied","Data":"eae0010720f20b0118950adac9f187d4fe021502a013f367411d233e5d95cbf6"} Sep 15 21:01:30.554178 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.554065 2580 scope.go:117] "RemoveContainer" containerID="85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492" Sep 15 21:01:30.554590 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.554564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "00f05f58-7565-43f5-916e-1b4476f14add" (UID: "00f05f58-7565-43f5-916e-1b4476f14add"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:01:30.554590 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.554581 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00f05f58-7565-43f5-916e-1b4476f14add" (UID: "00f05f58-7565-43f5-916e-1b4476f14add"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:01:30.554832 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.554808 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f05f58-7565-43f5-916e-1b4476f14add-kube-api-access-4lb2q" (OuterVolumeSpecName: "kube-api-access-4lb2q") pod "00f05f58-7565-43f5-916e-1b4476f14add" (UID: "00f05f58-7565-43f5-916e-1b4476f14add"). InnerVolumeSpecName "kube-api-access-4lb2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:01:30.567366 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.567341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:30.577034 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.577013 2580 scope.go:117] "RemoveContainer" containerID="85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492" Sep 15 21:01:30.577340 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:01:30.577319 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492\": container with ID starting with 85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492 not found: ID does not exist" containerID="85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492" Sep 15 21:01:30.577418 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.577350 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492"} err="failed to get container status \"85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492\": rpc error: code = NotFound desc = could not find container \"85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492\": container with ID starting with 85fb7ec8a70f7700e60575cb01738569b4681616e76c688a1968276bea301492 not found: ID does not exist" Sep 15 21:01:30.654297 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.654258 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:30.654588 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.654565 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f05f58-7565-43f5-916e-1b4476f14add-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:30.654721 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.654695 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f05f58-7565-43f5-916e-1b4476f14add-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:30.654829 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.654814 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lb2q\" (UniqueName: \"kubernetes.io/projected/00f05f58-7565-43f5-916e-1b4476f14add-kube-api-access-4lb2q\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:01:30.768755 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.768710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:30.870556 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.870525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:30.889086 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.889050 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8"] Sep 15 21:01:30.892721 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:30.892695 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-988a09-8648c878c7-d99f8"] Sep 15 21:01:31.316733 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:31.316695 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f05f58-7565-43f5-916e-1b4476f14add" path="/var/lib/kubelet/pods/00f05f58-7565-43f5-916e-1b4476f14add/volumes" Sep 15 21:01:31.627831 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:31.627746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:32.312205 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:32.312168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:33.104309 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:33.104272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:33.730958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:33.730917 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:34.351312 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:34.351276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:35.304608 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:35.304574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:35.359363 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:35.359330 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:35.930079 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:35.930044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:40.899220 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:40.899176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:42.017644 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:42.017609 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:42.118719 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:42.118675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:42.338094 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:42.338012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:42.840528 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:42.840494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:43.120094 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.119999 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn"] Sep 15 21:01:43.120762 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120731 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f05f58-7565-43f5-916e-1b4476f14add" containerName="maas-api" Sep 15 21:01:43.120762 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120755 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f05f58-7565-43f5-916e-1b4476f14add" containerName="maas-api" Sep 15 21:01:43.120967 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120773 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" Sep 15 21:01:43.120967 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120781 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" Sep 15 21:01:43.120967 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120809 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1486413c-178a-43b5-ba27-a39850380cad" containerName="test-curl-7752-27cff6" Sep 15 21:01:43.120967 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120818 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1486413c-178a-43b5-ba27-a39850380cad" containerName="test-curl-7752-27cff6" Sep 15 21:01:43.120967 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120962 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" Sep 15 21:01:43.121217 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120983 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f05f58-7565-43f5-916e-1b4476f14add" containerName="maas-api" Sep 15 21:01:43.121217 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.120995 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1486413c-178a-43b5-ba27-a39850380cad" containerName="test-curl-7752-27cff6" Sep 15 21:01:43.125850 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.125828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.127895 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.127876 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-280fb4-openshift-default-dockercfg-mkppl\"" Sep 15 21:01:43.134350 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.134326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn"] Sep 15 21:01:43.149355 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.149327 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:43.289885 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.289848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-podinfo\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290097 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.289929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-socket\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290097 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.289981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce890c77-7cf0-4bd6-9334-e45df65b141d-istiod-ca-cert\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290097 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.290025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-envoy\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290260 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.290104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-credential-socket\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290260 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.290146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-token\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290260 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.290187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-data\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290260 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.290236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-certs\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.290481 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.290279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgmlh\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-kube-api-access-kgmlh\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391046 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.390966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-podinfo\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-socket\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce890c77-7cf0-4bd6-9334-e45df65b141d-istiod-ca-cert\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-envoy\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-credential-socket\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-token\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-data\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391619 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-certs\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391619 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmlh\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-kube-api-access-kgmlh\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391619 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-socket\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391783 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-credential-socket\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391783 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391727 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-data\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.391894 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.391852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-certs\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.392239 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.392216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce890c77-7cf0-4bd6-9334-e45df65b141d-istiod-ca-cert\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.393438 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.393395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-envoy\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.393648 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.393632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-podinfo\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.398597 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.398565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-token\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.398688 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.398636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgmlh\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-kube-api-access-kgmlh\") pod \"e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.438346 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.438318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:43.721796 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.721763 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn"] Sep 15 21:01:43.723906 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:01:43.723878 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce890c77_7cf0_4bd6_9334_e45df65b141d.slice/crio-3668a83b3a1215a493dd1402a179e21ea0d47154daf24e79eff1cb5977c60e47 WatchSource:0}: Error finding container 3668a83b3a1215a493dd1402a179e21ea0d47154daf24e79eff1cb5977c60e47: Status 404 returned error can't find the container with id 3668a83b3a1215a493dd1402a179e21ea0d47154daf24e79eff1cb5977c60e47 Sep 15 21:01:43.726615 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.726580 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:01:43.726720 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.726647 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:01:43.726720 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:43.726676 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:01:44.086348 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:44.086309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:44.644790 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:44.644754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" event={"ID":"ce890c77-7cf0-4bd6-9334-e45df65b141d","Type":"ContainerStarted","Data":"3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3"} Sep 15 21:01:44.645173 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:44.644810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" event={"ID":"ce890c77-7cf0-4bd6-9334-e45df65b141d","Type":"ContainerStarted","Data":"3668a83b3a1215a493dd1402a179e21ea0d47154daf24e79eff1cb5977c60e47"} Sep 15 21:01:44.666252 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:44.666205 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" podStartSLOduration=1.666189288 podStartE2EDuration="1.666189288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:44.662768854 +0000 UTC m=+1493.792505440" watchObservedRunningTime="2026-09-15 21:01:44.666189288 +0000 UTC m=+1493.795925871" Sep 15 21:01:45.438758 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:45.438723 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:45.444030 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:45.444003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:45.650849 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:45.650811 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:45.651835 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:45.651816 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:01:45.832434 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:45.832372 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:46.543130 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:46.543082 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:46.638170 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:46.638118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:47.543261 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:47.543222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:48.325676 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:48.325640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:48.954494 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:48.954439 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:49.623216 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:49.623184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:50.238086 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.238056 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78"] Sep 15 21:01:50.238626 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.238611 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" Sep 15 21:01:50.238683 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.238628 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" Sep 15 21:01:50.238743 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.238734 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb07e50-b590-4a0c-8244-c45d83329b75" containerName="istio-proxy" Sep 15 21:01:50.242153 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.242132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.244190 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.244170 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-280fb4-serving-cert\"" Sep 15 21:01:50.252707 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.252666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78"] Sep 15 21:01:50.267664 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.267636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:50.322279 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.322235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:50.366821 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.366783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.366998 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.366834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztkm\" (UniqueName: \"kubernetes.io/projected/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-kube-api-access-bztkm\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.366998 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.366895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.367147 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.367035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.467808 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.467772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.468003 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.467848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztkm\" (UniqueName: \"kubernetes.io/projected/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-kube-api-access-bztkm\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.468003 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.467938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.468536 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.468509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.468699 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.468674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.468805 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:01:50.468579 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-serving-cert: secret "maas-api-e2e-shared-a-w3-280fb4-serving-cert" not found Sep 15 21:01:50.468856 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:01:50.468809 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls podName:f86fb478-25eb-4e4d-81f0-b3bd71f77bb4 nodeName:}" failed. No retries permitted until 2026-09-15 21:01:50.968786806 +0000 UTC m=+1500.098523367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls") pod "maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" (UID: "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4") : secret "maas-api-e2e-shared-a-w3-280fb4-serving-cert" not found Sep 15 21:01:50.470471 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.470451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.479237 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.479212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztkm\" (UniqueName: \"kubernetes.io/projected/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-kube-api-access-bztkm\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.682789 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.682743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:50.780963 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.780930 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:50.851323 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.851278 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:01:50.973997 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.973910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:50.976362 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:50.976340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:51.156426 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:51.156371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:51.398925 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:51.398890 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:01:51.405190 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:51.405168 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:01:51.435053 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:51.435029 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78"] Sep 15 21:01:51.435992 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:01:51.435963 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86fb478_25eb_4e4d_81f0_b3bd71f77bb4.slice/crio-ad012abc27494d40b44fc00597b9d7ae8f0145fb3c2a7d21a7dc51ae36404db1 WatchSource:0}: Error finding container ad012abc27494d40b44fc00597b9d7ae8f0145fb3c2a7d21a7dc51ae36404db1: Status 404 returned error can't find the container with id ad012abc27494d40b44fc00597b9d7ae8f0145fb3c2a7d21a7dc51ae36404db1 Sep 15 21:01:51.691184 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:51.691090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" event={"ID":"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4","Type":"ContainerStarted","Data":"ad012abc27494d40b44fc00597b9d7ae8f0145fb3c2a7d21a7dc51ae36404db1"} Sep 15 21:01:52.698905 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:52.698869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" event={"ID":"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4","Type":"ContainerStarted","Data":"7ae40698e5ddd6edc0836eaf8f672a0bd40332c25b394209d737b62fe66dddd3"} Sep 15 21:01:52.699335 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:52.698941 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:01:52.719935 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:52.719878 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" podStartSLOduration=2.3077936 podStartE2EDuration="2.719860692s" totalImagesPullingTime="412.067092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:52.716015233 +0000 UTC m=+1501.845751817" watchObservedRunningTime="2026-09-15 21:01:52.719860692 +0000 UTC m=+1501.849597277" Sep 15 21:01:58.711534 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:01:58.711496 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:02:04.171557 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:04.171521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:12.130861 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:12.130826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:14.168108 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:14.168016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:14.268820 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:14.268782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:14.421392 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:14.421265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:15.068266 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:15.068234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:15.839856 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:15.839804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:16.470821 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:16.470786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:16.665125 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:16.665081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:17.061959 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:17.061921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:17.364063 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:17.363938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:17.541584 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:17.541536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:19.747515 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:19.747468 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:20.432664 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:20.432623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:20.531981 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:20.531943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:21.342941 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:21.342909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:22.112702 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:22.112671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:23.054577 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.054487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:23.406032 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.405990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:23.907473 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.907437 2580 generic.go:356] "Generic (PLEG): container finished" podID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerID="3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3" exitCode=0 Sep 15 21:02:23.907663 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.907510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" event={"ID":"ce890c77-7cf0-4bd6-9334-e45df65b141d","Type":"ContainerDied","Data":"3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3"} Sep 15 21:02:23.907939 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.907925 2580 scope.go:117] "RemoveContainer" containerID="3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3" Sep 15 21:02:23.909215 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.909188 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:02:23.909291 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.909259 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:02:23.909327 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:23.909291 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:02:24.129034 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.128980 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:24.785322 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.785290 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm"] Sep 15 21:02:24.789241 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.789175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.791525 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.791500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-728877-serving-cert\"" Sep 15 21:02:24.798969 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.798942 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm"] Sep 15 21:02:24.815523 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.815374 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:24.849616 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.849325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhgsb\" (UniqueName: \"kubernetes.io/projected/3f216cfa-94c5-487a-bac0-61d7dee333fa-kube-api-access-vhgsb\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.849616 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.849427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.849616 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.849455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f216cfa-94c5-487a-bac0-61d7dee333fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.849616 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.849483 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.870555 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.870522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:24.932431 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.928438 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" event={"ID":"ce890c77-7cf0-4bd6-9334-e45df65b141d","Type":"ContainerStarted","Data":"80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18"} Sep 15 21:02:24.950454 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.950420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhgsb\" (UniqueName: \"kubernetes.io/projected/3f216cfa-94c5-487a-bac0-61d7dee333fa-kube-api-access-vhgsb\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.950676 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.950529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.950676 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.950555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f216cfa-94c5-487a-bac0-61d7dee333fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.950676 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.950602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.950869 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:02:24.950768 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-serving-cert: secret "maas-api-e2e-shared-b-w3-728877-serving-cert" not found Sep 15 21:02:24.950869 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:02:24.950849 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls podName:3f216cfa-94c5-487a-bac0-61d7dee333fa nodeName:}" failed. No retries permitted until 2026-09-15 21:02:25.450829184 +0000 UTC m=+1534.580565761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls") pod "maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" (UID: "3f216cfa-94c5-487a-bac0-61d7dee333fa") : secret "maas-api-e2e-shared-b-w3-728877-serving-cert" not found Sep 15 21:02:24.951363 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.951343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f216cfa-94c5-487a-bac0-61d7dee333fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.953048 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.953032 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:24.958332 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:24.958311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhgsb\" (UniqueName: \"kubernetes.io/projected/3f216cfa-94c5-487a-bac0-61d7dee333fa-kube-api-access-vhgsb\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:25.061282 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.061191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:25.269487 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.269437 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:25.354131 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.354046 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:25.439439 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.438850 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:02:25.439439 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.438975 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 15 21:02:25.439439 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.439017 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 15 21:02:25.455213 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.455188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:25.457701 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.457681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:25.703283 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:25.703191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:26.005309 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:26.005282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm"] Sep 15 21:02:26.006578 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:02:26.006554 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f216cfa_94c5_487a_bac0_61d7dee333fa.slice/crio-82561cc8568612b2dceb89f0f633c04c632cf4731e07b75f881b6a9e5bf7057b WatchSource:0}: Error finding container 82561cc8568612b2dceb89f0f633c04c632cf4731e07b75f881b6a9e5bf7057b: Status 404 returned error can't find the container with id 82561cc8568612b2dceb89f0f633c04c632cf4731e07b75f881b6a9e5bf7057b Sep 15 21:02:26.439440 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:26.439340 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 15 21:02:26.439823 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:26.439438 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 15 21:02:26.948238 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:26.948203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" event={"ID":"3f216cfa-94c5-487a-bac0-61d7dee333fa","Type":"ContainerStarted","Data":"bb6f51f5980070eb00607216c601807065c67a241b9d717d554a3c347957ae23"} Sep 15 21:02:26.948472 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:26.948268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" event={"ID":"3f216cfa-94c5-487a-bac0-61d7dee333fa","Type":"ContainerStarted","Data":"82561cc8568612b2dceb89f0f633c04c632cf4731e07b75f881b6a9e5bf7057b"} Sep 15 21:02:26.948472 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:26.948430 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:26.966038 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:26.965987 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" podStartSLOduration=2.5155481159999997 podStartE2EDuration="2.965971025s" totalImagesPullingTime="450.422909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:26.963512803 +0000 UTC m=+1536.093249393" watchObservedRunningTime="2026-09-15 21:02:26.965971025 +0000 UTC m=+1536.095707642" Sep 15 21:02:27.439449 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:27.439380 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 15 21:02:27.439945 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:27.439468 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 15 21:02:28.439384 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:28.439341 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 15 21:02:28.439600 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:28.439431 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 15 21:02:29.439163 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:29.439117 2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 15 21:02:29.439367 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:29.439187 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 15 21:02:30.443606 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:30.443568 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:02:30.444023 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:30.443863 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:02:30.444632 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:30.444609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:02:32.959840 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:32.959808 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:02:47.698500 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:47.698452 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:48.334912 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:48.334871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:49.287095 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:49.287060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:50.009481 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:50.009446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:50.495040 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:50.494944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:50.594892 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:50.594855 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:51.257304 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:51.257270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:51.505128 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:51.505100 2580 scope.go:117] "RemoveContainer" containerID="552a309d789bd0477774ca637691f79e83b69f4be4e97580ee5c4f9f32ce8887" Sep 15 21:02:51.946235 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:51.946145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:02:52.614028 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:02:52.613995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:15.470331 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:15.470297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:15.524551 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:15.524514 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:16.398771 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:16.398734 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:34.192874 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:34.192838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:36.895306 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:36.895272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:53.007526 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:53.007487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:53.691395 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:53.691357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:54.530955 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:54.530914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:55.322933 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:55.322900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:55.836596 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:55.836559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:56.628286 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:56.628243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:57.261635 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:57.261602 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:58.001317 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:58.001276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:03:58.977511 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:03:58.977477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:08.267778 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:08.267738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:08.365457 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:08.365424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:08.467155 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:08.466885 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:08.531810 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:08.531724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:08.940083 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:08.939997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:09.748501 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:09.748455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:10.445019 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:10.444982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:11.147452 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:11.147419 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:11.967523 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:11.967478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:12.154094 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:12.154044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:12.800650 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:12.800541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:13.624133 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:13.624092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:16.189636 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:16.189599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:17.245254 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:17.245221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:37.342158 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:37.341974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:38.469610 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:38.469577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:47.729764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:47.729724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:52.358599 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:52.358564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:53.695366 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:53.695328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:54.447124 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:54.447090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:04:59.914135 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:04:59.914069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:04.064599 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:04.064553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:27.924067 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:27.923950 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:31.728273 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:31.728229 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:31.824809 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:31.824771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:32.711247 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:32.711208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:33.468701 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:33.468591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:33.553135 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:33.553101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:33.653921 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:33.653874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:34.402133 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:34.402101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:34.498947 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:34.498912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:34.604848 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:34.604805 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:35.933477 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:35.933432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:41.706567 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:41.706527 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:41.806276 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:41.806240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:42.415201 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:42.415169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:48.585435 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:48.585385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:48.696497 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:48.696459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:48.790266 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:48.790207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:59.353638 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:59.353599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:05:59.588297 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:05:59.588259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:03.268010 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:03.267941 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:12.027342 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:12.027284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:20.620934 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:20.620897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:23.584762 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:23.584725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:26.393285 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:26.393248 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:26.493982 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:26.493944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:26.776087 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:26.776055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:27.095601 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:27.095524 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:28.016063 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:28.016020 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:29.653360 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:29.653320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:29.899811 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:29.899719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:29.988801 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:29.988715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:30.509109 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:30.509074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:30.839486 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:30.839375 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:31.335681 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:31.335646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:32.000635 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:32.000597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:32.733601 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:32.733564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:33.407253 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:33.407214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:34.110485 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:34.110447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:34.212806 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:34.212770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:34.351616 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:34.351582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:34.680613 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:34.680572 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:34.777066 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:34.777032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:51.462399 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:51.462365 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:06:51.466742 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:51.466713 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:06:56.141449 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:56.141396 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:57.307054 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:57.306987 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:57.622486 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:57.622382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:57.820755 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:57.820718 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:58.535569 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:58.535539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:59.267324 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:59.267291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:06:59.929282 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:06:59.929244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:00.080010 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:00.079976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:00.570084 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:00.570050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:01.586882 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:01.586847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:01.686896 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:01.686860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:03.523090 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:03.523049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:04.065739 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:04.065706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:04.170039 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:04.170005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:04.381155 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:04.381067 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:05.345775 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:05.345740 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:06.034675 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:06.034643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:06.752023 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:06.751989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:07.537290 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:07.537228 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:08.314876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:08.314842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:08.994082 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:08.994045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:09.907491 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:09.907450 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm"] Sep 15 21:07:09.912518 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:09.912493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:09.917475 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:09.917451 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-14aeb7-serving-cert\"" Sep 15 21:07:09.921283 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:09.921261 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm"] Sep 15 21:07:09.937545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:09.937516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:10.000892 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.000854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:10.095773 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.095719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-maas-api-tls\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.095773 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.095786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-metrics-certs\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.096323 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.095859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-openshift-service-ca\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.096323 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.096008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqfz\" (UniqueName: \"kubernetes.io/projected/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-kube-api-access-cwqfz\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.100240 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.100216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:10.197460 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.197347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-openshift-service-ca\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.197613 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.197470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqfz\" (UniqueName: \"kubernetes.io/projected/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-kube-api-access-cwqfz\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.197613 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.197595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-maas-api-tls\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.197691 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.197624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-metrics-certs\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.198184 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.198160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-openshift-service-ca\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.200225 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.200201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-metrics-certs\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.200326 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.200295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-maas-api-tls\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.204974 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.204949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqfz\" (UniqueName: \"kubernetes.io/projected/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-kube-api-access-cwqfz\") pod \"maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.230548 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.230516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:10.385739 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.385705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:10.482053 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.481972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:10.518765 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.518738 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm"] Sep 15 21:07:10.520774 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:07:10.520736 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d818ed2_4ec8_4cf6_a46a_d36bcd21ef48.slice/crio-f4777062e94a6f62361367050a55f431032436a88130e84836267e5518c743e9 WatchSource:0}: Error finding container f4777062e94a6f62361367050a55f431032436a88130e84836267e5518c743e9: Status 404 returned error can't find the container with id f4777062e94a6f62361367050a55f431032436a88130e84836267e5518c743e9 Sep 15 21:07:10.522289 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.522269 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:07:10.826707 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:10.826674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" event={"ID":"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48","Type":"ContainerStarted","Data":"f4777062e94a6f62361367050a55f431032436a88130e84836267e5518c743e9"} Sep 15 21:07:11.537756 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:11.537721 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:12.841363 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:12.841320 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" event={"ID":"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48","Type":"ContainerStarted","Data":"a0107ac31ce41d1867fd150f5cf6e6df84112e1a5b94691fd3b14e1af8dc5fef"} Sep 15 21:07:12.841838 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:12.841389 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:12.862431 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:12.862367 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" podStartSLOduration=2.273573731 podStartE2EDuration="3.862350614s" totalImagesPullingTime="1.588776883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:07:12.8586199 +0000 UTC m=+1821.988356485" watchObservedRunningTime="2026-09-15 21:07:12.862350614 +0000 UTC m=+1821.992087197" Sep 15 21:07:18.855145 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:18.855110 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:07:30.796479 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:30.796441 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:33.971553 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:33.971522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:34.208445 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:34.208395 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:34.309781 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:34.309683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:34.938760 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:34.938724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:35.614657 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:35.614615 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:36.343624 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:36.343587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:37.124893 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:37.124853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:37.982178 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:37.982146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:38.818086 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:38.818050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:39.591007 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:39.590969 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:40.434686 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:40.434647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:41.340962 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:41.340921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:42.021512 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:42.021460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:42.712872 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:42.712831 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:43.500898 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:43.500848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:44.153151 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:44.153119 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:47.217056 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:47.217013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:52.787308 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:52.787272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:52.883941 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:52.883906 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:52.974315 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:52.974285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:53.036832 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:53.036793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:53.327292 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:53.327260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:53.481886 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:53.481852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:54.136192 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:54.136157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:54.932579 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:54.932541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:55.597815 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:55.597778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:56.410660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:56.410610 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:57.070710 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:57.070674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:57.754876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:57.754843 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:58.528262 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:58.528231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:59.642675 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:59.642639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:59.741640 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:59.741599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:07:59.888459 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:07:59.888425 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:00.519835 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:00.519802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:00.619306 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:00.619267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:01.580316 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:01.580279 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:02.334713 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:02.334676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:03.113205 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:03.113157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:03.833704 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:03.833669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:04.744507 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:04.744462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:05.559346 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:05.559313 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg"] Sep 15 21:08:05.682664 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:05.682599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:06.287212 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:06.287172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:06.691837 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:06.691756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:07.355783 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:07.355748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:10.222488 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:10.222453 2580 generic.go:356] "Generic (PLEG): container finished" podID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerID="031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3" exitCode=137 Sep 15 21:08:10.222914 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:10.222528 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" event={"ID":"2cf11874-73cc-4d79-9d9e-afbe42301acf","Type":"ContainerDied","Data":"031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3"} Sep 15 21:08:10.222957 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:10.222896 2580 scope.go:117] "RemoveContainer" containerID="031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3" Sep 15 21:08:11.231378 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:11.231339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" event={"ID":"2cf11874-73cc-4d79-9d9e-afbe42301acf","Type":"ContainerStarted","Data":"4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770"} Sep 15 21:08:11.231794 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:11.231582 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 21:08:15.872767 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:15.872677 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p"] Sep 15 21:08:15.873219 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:15.872886 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" podUID="aa3d2622-0cac-4f1a-86fa-20a6613e4811" containerName="maas-api" containerID="cri-o://79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc" gracePeriod=30 Sep 15 21:08:15.900667 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:15.900636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:15.948359 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:15.948326 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw"] Sep 15 21:08:15.948627 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:15.948595 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" podUID="0180a0cf-01c3-4c36-8f7a-485107ebd453" containerName="payload-pre-processing" containerID="cri-o://a7a4e62f086523c2cc903d17e3c2238f1ac1dd6828f5b0943bcb8885dfc3e692" gracePeriod=30 Sep 15 21:08:16.000944 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.000911 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.037620 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.037583 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm"] Sep 15 21:08:16.037870 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.037840 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" podUID="0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" containerName="maas-api" containerID="cri-o://a0107ac31ce41d1867fd150f5cf6e6df84112e1a5b94691fd3b14e1af8dc5fef" gracePeriod=30 Sep 15 21:08:16.108666 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.108594 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.202387 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.202344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.257643 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.257613 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 21:08:16.269260 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.269149 2580 generic.go:356] "Generic (PLEG): container finished" podID="0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" containerID="a0107ac31ce41d1867fd150f5cf6e6df84112e1a5b94691fd3b14e1af8dc5fef" exitCode=0 Sep 15 21:08:16.269260 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.269219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" event={"ID":"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48","Type":"ContainerDied","Data":"a0107ac31ce41d1867fd150f5cf6e6df84112e1a5b94691fd3b14e1af8dc5fef"} Sep 15 21:08:16.271158 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.271124 2580 generic.go:356] "Generic (PLEG): container finished" podID="0180a0cf-01c3-4c36-8f7a-485107ebd453" containerID="a7a4e62f086523c2cc903d17e3c2238f1ac1dd6828f5b0943bcb8885dfc3e692" exitCode=0 Sep 15 21:08:16.271293 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.271268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" event={"ID":"0180a0cf-01c3-4c36-8f7a-485107ebd453","Type":"ContainerDied","Data":"a7a4e62f086523c2cc903d17e3c2238f1ac1dd6828f5b0943bcb8885dfc3e692"} Sep 15 21:08:16.272890 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.272870 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa3d2622-0cac-4f1a-86fa-20a6613e4811" containerID="79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc" exitCode=0 Sep 15 21:08:16.273005 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.272921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" event={"ID":"aa3d2622-0cac-4f1a-86fa-20a6613e4811","Type":"ContainerDied","Data":"79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc"} Sep 15 21:08:16.273005 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.272926 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" Sep 15 21:08:16.273005 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.272970 2580 scope.go:117] "RemoveContainer" containerID="79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc" Sep 15 21:08:16.273156 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.272959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p" event={"ID":"aa3d2622-0cac-4f1a-86fa-20a6613e4811","Type":"ContainerDied","Data":"3f520b05f84b577b35dd1df002b5e4871fb42ae30f3e3472c9200072807b6e70"} Sep 15 21:08:16.291052 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.290779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.293539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.293518 2580 scope.go:117] "RemoveContainer" containerID="79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc" Sep 15 21:08:16.293986 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:08:16.293936 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc\": container with ID starting with 79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc not found: ID does not exist" containerID="79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc" Sep 15 21:08:16.294096 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.293999 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc"} err="failed to get container status \"79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc\": rpc error: code = NotFound desc = could not find container \"79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc\": container with ID starting with 79135d529c40232209ce7972633ca3990181914562d9be1f936b61c0750edffc not found: ID does not exist" Sep 15 21:08:16.296463 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.296430 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 21:08:16.302964 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.302944 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-metrics-certs\") pod \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " Sep 15 21:08:16.303075 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.302984 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-maas-api-tls\") pod \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " Sep 15 21:08:16.303075 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.303067 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a0cf-01c3-4c36-8f7a-485107ebd453-plugins-config-volume\") pod \"0180a0cf-01c3-4c36-8f7a-485107ebd453\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " Sep 15 21:08:16.303196 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.303103 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3d2622-0cac-4f1a-86fa-20a6613e4811-openshift-service-ca\") pod \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " Sep 15 21:08:16.303196 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.303137 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbbsx\" (UniqueName: \"kubernetes.io/projected/0180a0cf-01c3-4c36-8f7a-485107ebd453-kube-api-access-bbbsx\") pod \"0180a0cf-01c3-4c36-8f7a-485107ebd453\" (UID: \"0180a0cf-01c3-4c36-8f7a-485107ebd453\") " Sep 15 21:08:16.303600 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.303505 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3d2622-0cac-4f1a-86fa-20a6613e4811-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa3d2622-0cac-4f1a-86fa-20a6613e4811" (UID: "aa3d2622-0cac-4f1a-86fa-20a6613e4811"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:16.303600 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.303564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0180a0cf-01c3-4c36-8f7a-485107ebd453-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0180a0cf-01c3-4c36-8f7a-485107ebd453" (UID: "0180a0cf-01c3-4c36-8f7a-485107ebd453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:16.303767 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.303664 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a0cf-01c3-4c36-8f7a-485107ebd453-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.303767 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.303687 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3d2622-0cac-4f1a-86fa-20a6613e4811-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.306688 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.306654 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0180a0cf-01c3-4c36-8f7a-485107ebd453-kube-api-access-bbbsx" (OuterVolumeSpecName: "kube-api-access-bbbsx") pod "0180a0cf-01c3-4c36-8f7a-485107ebd453" (UID: "0180a0cf-01c3-4c36-8f7a-485107ebd453"). InnerVolumeSpecName "kube-api-access-bbbsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:16.306688 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.306663 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa3d2622-0cac-4f1a-86fa-20a6613e4811" (UID: "aa3d2622-0cac-4f1a-86fa-20a6613e4811"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:16.306851 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.306710 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa3d2622-0cac-4f1a-86fa-20a6613e4811" (UID: "aa3d2622-0cac-4f1a-86fa-20a6613e4811"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:16.347356 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.347328 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:08:16.404906 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.404822 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2stw\" (UniqueName: \"kubernetes.io/projected/aa3d2622-0cac-4f1a-86fa-20a6613e4811-kube-api-access-m2stw\") pod \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\" (UID: \"aa3d2622-0cac-4f1a-86fa-20a6613e4811\") " Sep 15 21:08:16.404906 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.404881 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-maas-api-tls\") pod \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " Sep 15 21:08:16.405101 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.404936 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-openshift-service-ca\") pod \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " Sep 15 21:08:16.405101 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.404972 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwqfz\" (UniqueName: \"kubernetes.io/projected/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-kube-api-access-cwqfz\") pod \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " Sep 15 21:08:16.405101 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.405013 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-metrics-certs\") pod \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\" (UID: \"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48\") " Sep 15 21:08:16.405375 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.405341 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" (UID: "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:16.405545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.405465 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.405545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.405486 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3d2622-0cac-4f1a-86fa-20a6613e4811-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.405545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.405498 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbbsx\" (UniqueName: \"kubernetes.io/projected/0180a0cf-01c3-4c36-8f7a-485107ebd453-kube-api-access-bbbsx\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.407205 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.407171 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-kube-api-access-cwqfz" (OuterVolumeSpecName: "kube-api-access-cwqfz") pod "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" (UID: "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48"). InnerVolumeSpecName "kube-api-access-cwqfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:16.407205 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.407195 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" (UID: "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:16.407342 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.407182 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3d2622-0cac-4f1a-86fa-20a6613e4811-kube-api-access-m2stw" (OuterVolumeSpecName: "kube-api-access-m2stw") pod "aa3d2622-0cac-4f1a-86fa-20a6613e4811" (UID: "aa3d2622-0cac-4f1a-86fa-20a6613e4811"). InnerVolumeSpecName "kube-api-access-m2stw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:16.407342 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.407241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" (UID: "0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:16.422791 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.422755 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.507630 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.507518 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.507630 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.507558 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqfz\" (UniqueName: \"kubernetes.io/projected/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-kube-api-access-cwqfz\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.507630 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.507575 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.507630 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.507588 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2stw\" (UniqueName: \"kubernetes.io/projected/aa3d2622-0cac-4f1a-86fa-20a6613e4811-kube-api-access-m2stw\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.507630 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.507601 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:16.511785 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.511655 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.605677 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.605648 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p"] Sep 15 21:08:16.614718 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.614687 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3f1c93-69cdb65d7-qc28p"] Sep 15 21:08:16.615500 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.615465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.692766 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.692680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:16.943724 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:16.943639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:17.282223 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.282189 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" Sep 15 21:08:17.282223 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.282214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw" event={"ID":"0180a0cf-01c3-4c36-8f7a-485107ebd453","Type":"ContainerDied","Data":"ac752d75ca5dd115306d452afd9a7f862fa3854fda3782ad4bf7e9033535fc57"} Sep 15 21:08:17.282483 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.282309 2580 scope.go:117] "RemoveContainer" containerID="a7a4e62f086523c2cc903d17e3c2238f1ac1dd6828f5b0943bcb8885dfc3e692" Sep 15 21:08:17.285174 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.285150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" event={"ID":"0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48","Type":"ContainerDied","Data":"f4777062e94a6f62361367050a55f431032436a88130e84836267e5518c743e9"} Sep 15 21:08:17.285298 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.285182 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm" Sep 15 21:08:17.293466 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.293448 2580 scope.go:117] "RemoveContainer" containerID="a0107ac31ce41d1867fd150f5cf6e6df84112e1a5b94691fd3b14e1af8dc5fef" Sep 15 21:08:17.314772 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.314743 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3d2622-0cac-4f1a-86fa-20a6613e4811" path="/var/lib/kubelet/pods/aa3d2622-0cac-4f1a-86fa-20a6613e4811/volumes" Sep 15 21:08:17.319773 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.319752 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw"] Sep 15 21:08:17.325691 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.325669 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3f1c93-d6f659cfd-p44gw"] Sep 15 21:08:17.337869 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.337847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm"] Sep 15 21:08:17.340996 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.340972 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14aeb7-599c8dbc69-8cwgm"] Sep 15 21:08:17.670316 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:17.670227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:18.413709 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:18.413673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:19.267856 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:19.267825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:19.315569 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:19.315529 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0180a0cf-01c3-4c36-8f7a-485107ebd453" path="/var/lib/kubelet/pods/0180a0cf-01c3-4c36-8f7a-485107ebd453/volumes" Sep 15 21:08:19.316356 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:19.316317 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" path="/var/lib/kubelet/pods/0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48/volumes" Sep 15 21:08:20.124944 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.124891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:20.579382 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.579340 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" podUID="3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" containerName="main" containerID="cri-o://c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc" gracePeriod=45 Sep 15 21:08:20.835215 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.835142 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 21:08:20.836227 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.836203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:20.857668 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.857636 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-home\") pod \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " Sep 15 21:08:20.857808 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.857706 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rj6p\" (UniqueName: \"kubernetes.io/projected/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-kube-api-access-2rj6p\") pod \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " Sep 15 21:08:20.857808 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.857786 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-model-cache\") pod \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " Sep 15 21:08:20.857887 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.857828 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tls-certs\") pod \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " Sep 15 21:08:20.857887 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.857865 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-dshm\") pod \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " Sep 15 21:08:20.857980 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.857926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-home" (OuterVolumeSpecName: "home") pod "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" (UID: "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.857980 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.857938 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tmp-dir\") pod \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\" (UID: \"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c\") " Sep 15 21:08:20.858178 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.858155 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" (UID: "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.858726 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.858374 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" (UID: "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.858726 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.858498 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.858726 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.858516 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.858726 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.858557 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.860788 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.860751 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" (UID: "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:20.861020 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.861000 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-dshm" (OuterVolumeSpecName: "dshm") pod "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" (UID: "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:20.861083 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.861025 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-kube-api-access-2rj6p" (OuterVolumeSpecName: "kube-api-access-2rj6p") pod "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" (UID: "3f9bcad3-3d3c-46c9-a855-1b739d78cf8c"). InnerVolumeSpecName "kube-api-access-2rj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:20.959804 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.959766 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.959804 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.959796 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:20.959804 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:20.959809 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rj6p\" (UniqueName: \"kubernetes.io/projected/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c-kube-api-access-2rj6p\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:21.322514 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.322465 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" containerID="c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc" exitCode=0 Sep 15 21:08:21.322892 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.322801 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" Sep 15 21:08:21.323668 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.323637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" event={"ID":"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c","Type":"ContainerDied","Data":"c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc"} Sep 15 21:08:21.323668 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.323667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg" event={"ID":"3f9bcad3-3d3c-46c9-a855-1b739d78cf8c","Type":"ContainerDied","Data":"ee90c0043557a8ad32f27e61f91fd1562606fa5e99049e39733b22fd65321041"} Sep 15 21:08:21.323856 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.323685 2580 scope.go:117] "RemoveContainer" containerID="c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc" Sep 15 21:08:21.334227 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.334210 2580 scope.go:117] "RemoveContainer" containerID="c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc" Sep 15 21:08:21.334517 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:08:21.334498 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc\": container with ID starting with c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc not found: ID does not exist" containerID="c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc" Sep 15 21:08:21.334605 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.334524 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc"} err="failed to get container status \"c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc\": rpc error: code = NotFound desc = could not find container \"c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc\": container with ID starting with c7ba3c67d3893adeb9e518e949d008537c6fb25c9351d1a4708368d7c8043fcc not found: ID does not exist" Sep 15 21:08:21.354290 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.354265 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg"] Sep 15 21:08:21.364581 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.364558 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/premium-w0-3f1c93-kserve-d9f66cdcf-b8pgg"] Sep 15 21:08:21.703877 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:21.703796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:22.240449 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:22.240415 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 21:08:23.314746 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:23.314705 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" path="/var/lib/kubelet/pods/3f9bcad3-3d3c-46c9-a855-1b739d78cf8c/volumes" Sep 15 21:08:24.168183 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:24.168148 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm"] Sep 15 21:08:24.168370 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:24.168340 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" containerID="cri-o://45cfe7e31fa783b8ac88a920e2a1323d38c7aa7b395b6a45098bbfbcdea6c418" gracePeriod=30 Sep 15 21:08:24.304131 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:24.304091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:24.416698 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:24.416668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:25.273422 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:25.273378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:25.372670 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:25.372635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:25.817476 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:25.817420 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:26.718772 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:26.718731 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:27.533271 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:27.533231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:28.237145 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:28.237109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:28.822698 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:28.822667 2580 patch_prober.go:28] interesting pod/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:08:28.823092 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:28.822733 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:08:28.959471 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:28.959433 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:29.377496 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.377464 2580 generic.go:356] "Generic (PLEG): container finished" podID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerID="45cfe7e31fa783b8ac88a920e2a1323d38c7aa7b395b6a45098bbfbcdea6c418" exitCode=0 Sep 15 21:08:29.377653 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.377515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" event={"ID":"a1b4b1ec-f9af-4b73-a2de-60b17235156f","Type":"ContainerDied","Data":"45cfe7e31fa783b8ac88a920e2a1323d38c7aa7b395b6a45098bbfbcdea6c418"} Sep 15 21:08:29.377653 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.377567 2580 scope.go:117] "RemoveContainer" containerID="77e54363eff6eaf37500c22c103b20a49afe83fe9fd505717dd3ef7801dd9e01" Sep 15 21:08:29.434231 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.434171 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 21:08:29.557984 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.557953 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-token\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558166 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558001 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-envoy\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558166 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558027 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istiod-ca-cert\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558300 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-credential-socket\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558300 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558208 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-data\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558426 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558376 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-podinfo\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558495 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558440 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-socket\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558495 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558472 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-certs\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558500 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp85p\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-kube-api-access-sp85p\") pod \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\" (UID: \"a1b4b1ec-f9af-4b73-a2de-60b17235156f\") " Sep 15 21:08:29.558611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558498 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:29.558611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558540 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:29.558783 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558726 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:29.558783 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558744 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:29.558783 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558758 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:29.558989 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558968 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.559071 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.558997 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.559071 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.559013 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.559071 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.559029 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.559071 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.559045 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.560204 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.560174 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:29.560323 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.560224 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:29.560754 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.560728 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-kube-api-access-sp85p" (OuterVolumeSpecName: "kube-api-access-sp85p") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "kube-api-access-sp85p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:29.560938 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.560917 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1b4b1ec-f9af-4b73-a2de-60b17235156f" (UID: "a1b4b1ec-f9af-4b73-a2de-60b17235156f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:08:29.653105 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.653058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:29.660447 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.660398 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.660447 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.660445 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp85p\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-kube-api-access-sp85p\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.660611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.660455 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:29.660611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:29.660463 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1b4b1ec-f9af-4b73-a2de-60b17235156f-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:30.384921 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:30.384893 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" Sep 15 21:08:30.385376 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:30.384931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm" event={"ID":"a1b4b1ec-f9af-4b73-a2de-60b17235156f","Type":"ContainerDied","Data":"066ea002ed0cde204aa3b88fd466097f596303fc20d15226b0f92e2473888d75"} Sep 15 21:08:30.385376 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:30.385008 2580 scope.go:117] "RemoveContainer" containerID="45cfe7e31fa783b8ac88a920e2a1323d38c7aa7b395b6a45098bbfbcdea6c418" Sep 15 21:08:30.410937 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:30.410905 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm"] Sep 15 21:08:30.413760 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:30.413739 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3f1c93-openshift-default-786966694-b5gsm"] Sep 15 21:08:30.819815 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:30.819773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:31.316398 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:31.316356 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" path="/var/lib/kubelet/pods/a1b4b1ec-f9af-4b73-a2de-60b17235156f/volumes" Sep 15 21:08:35.941701 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:35.941668 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q"] Sep 15 21:08:35.942203 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:35.941869 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" podUID="7e1af04f-1277-4565-8f0b-344bce74c765" containerName="maas-api" containerID="cri-o://5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17" gracePeriod=30 Sep 15 21:08:35.976700 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:35.976672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:36.225154 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.224832 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 21:08:36.337214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.337169 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1af04f-1277-4565-8f0b-344bce74c765-openshift-service-ca\") pod \"7e1af04f-1277-4565-8f0b-344bce74c765\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " Sep 15 21:08:36.337430 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.337243 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-metrics-certs\") pod \"7e1af04f-1277-4565-8f0b-344bce74c765\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " Sep 15 21:08:36.337430 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.337292 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls\") pod \"7e1af04f-1277-4565-8f0b-344bce74c765\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " Sep 15 21:08:36.337430 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.337313 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxz26\" (UniqueName: \"kubernetes.io/projected/7e1af04f-1277-4565-8f0b-344bce74c765-kube-api-access-gxz26\") pod \"7e1af04f-1277-4565-8f0b-344bce74c765\" (UID: \"7e1af04f-1277-4565-8f0b-344bce74c765\") " Sep 15 21:08:36.337703 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.337670 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1af04f-1277-4565-8f0b-344bce74c765-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e1af04f-1277-4565-8f0b-344bce74c765" (UID: "7e1af04f-1277-4565-8f0b-344bce74c765"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:36.339495 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.339468 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7e1af04f-1277-4565-8f0b-344bce74c765" (UID: "7e1af04f-1277-4565-8f0b-344bce74c765"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:36.339605 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.339496 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e1af04f-1277-4565-8f0b-344bce74c765" (UID: "7e1af04f-1277-4565-8f0b-344bce74c765"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:36.339605 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.339544 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1af04f-1277-4565-8f0b-344bce74c765-kube-api-access-gxz26" (OuterVolumeSpecName: "kube-api-access-gxz26") pod "7e1af04f-1277-4565-8f0b-344bce74c765" (UID: "7e1af04f-1277-4565-8f0b-344bce74c765"). InnerVolumeSpecName "kube-api-access-gxz26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:36.426028 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.425989 2580 generic.go:356] "Generic (PLEG): container finished" podID="7e1af04f-1277-4565-8f0b-344bce74c765" containerID="5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17" exitCode=0 Sep 15 21:08:36.426213 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.426044 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" Sep 15 21:08:36.426213 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.426072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" event={"ID":"7e1af04f-1277-4565-8f0b-344bce74c765","Type":"ContainerDied","Data":"5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17"} Sep 15 21:08:36.426213 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.426140 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q" event={"ID":"7e1af04f-1277-4565-8f0b-344bce74c765","Type":"ContainerDied","Data":"42f8edaeaac09021d1ba90b0160542bc560dcb78265e214cfb3f20fc3b42c683"} Sep 15 21:08:36.426213 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.426173 2580 scope.go:117] "RemoveContainer" containerID="5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17" Sep 15 21:08:36.437081 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.437065 2580 scope.go:117] "RemoveContainer" containerID="5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17" Sep 15 21:08:36.437355 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:08:36.437336 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17\": container with ID starting with 5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17 not found: ID does not exist" containerID="5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17" Sep 15 21:08:36.437430 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.437364 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17"} err="failed to get container status \"5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17\": rpc error: code = NotFound desc = could not find container \"5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17\": container with ID starting with 5276d86399050689d58893def765f342f048e1076581bfa75afd87a6ab576f17 not found: ID does not exist" Sep 15 21:08:36.438439 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.438392 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1af04f-1277-4565-8f0b-344bce74c765-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:36.438525 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.438445 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:36.438525 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.438460 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e1af04f-1277-4565-8f0b-344bce74c765-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:36.438525 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.438470 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxz26\" (UniqueName: \"kubernetes.io/projected/7e1af04f-1277-4565-8f0b-344bce74c765-kube-api-access-gxz26\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:36.448549 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.448499 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q"] Sep 15 21:08:36.451908 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.451885 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de4215-7c9b994696-qf27q"] Sep 15 21:08:36.567666 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.567629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:36.667156 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.667122 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:36.767311 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:36.767282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:37.315778 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:37.315745 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1af04f-1277-4565-8f0b-344bce74c765" path="/var/lib/kubelet/pods/7e1af04f-1277-4565-8f0b-344bce74c765/volumes" Sep 15 21:08:37.529200 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:37.529164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:38.267013 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:38.266973 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:39.110126 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:39.110093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:39.861469 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:39.861433 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:43.084785 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.084752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:43.186130 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.186093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:43.285306 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.285270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:43.387335 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.387256 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:43.488764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.488726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:43.588570 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.588538 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:43.738479 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.738357 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75"] Sep 15 21:08:43.738650 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:43.738592 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" containerID="cri-o://04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1" gracePeriod=30 Sep 15 21:08:44.067325 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:44.067291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:44.401435 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:44.401345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:45.279564 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:45.279531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:45.926728 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:45.926687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:46.831331 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:46.831290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:47.523654 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:47.523616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:48.315256 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:48.315222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:48.998227 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:48.998203 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 21:08:49.067496 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067465 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-socket\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.067671 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067576 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-data\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.067671 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067628 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-podinfo\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.067806 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-credential-socket\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.067806 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067729 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:49.067806 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067745 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fsdf\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-kube-api-access-2fsdf\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.067966 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-token\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.067966 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067874 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:49.067966 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067900 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-certs\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.067966 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067946 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istiod-ca-cert\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.068171 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067973 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-envoy\") pod \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\" (UID: \"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce\") " Sep 15 21:08:49.068171 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.067970 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:49.068268 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.068174 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:49.068491 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.068462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:49.068491 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.068481 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.068603 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.068504 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.068603 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.068518 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.068603 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.068530 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.070085 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.070064 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:08:49.070393 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.070366 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:08:49.070488 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.070432 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:49.070488 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.070463 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-kube-api-access-2fsdf" (OuterVolumeSpecName: "kube-api-access-2fsdf") pod "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" (UID: "0817b405-ca0c-4ec7-bdd0-71dc6337f7ce"). InnerVolumeSpecName "kube-api-access-2fsdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:49.169214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.169176 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fsdf\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-kube-api-access-2fsdf\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.169214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.169207 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.169214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.169216 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.169443 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.169224 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.169443 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.169232 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:49.512477 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.512439 2580 generic.go:356] "Generic (PLEG): container finished" podID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerID="04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1" exitCode=0 Sep 15 21:08:49.512653 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.512504 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" Sep 15 21:08:49.512653 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.512528 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" event={"ID":"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce","Type":"ContainerDied","Data":"04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1"} Sep 15 21:08:49.512653 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.512589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75" event={"ID":"0817b405-ca0c-4ec7-bdd0-71dc6337f7ce","Type":"ContainerDied","Data":"96b58c2f0969a7cf8029f98b36764903ba4cd15c7cd3bff062a9d4e352ac74d2"} Sep 15 21:08:49.512653 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.512619 2580 scope.go:117] "RemoveContainer" containerID="04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1" Sep 15 21:08:49.522987 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.522969 2580 scope.go:117] "RemoveContainer" containerID="49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6" Sep 15 21:08:49.530728 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.530696 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75"] Sep 15 21:08:49.534389 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.534364 2580 scope.go:117] "RemoveContainer" containerID="04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1" Sep 15 21:08:49.534880 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:08:49.534823 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1\": container with ID starting with 04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1 not found: ID does not exist" containerID="04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1" Sep 15 21:08:49.534880 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.534858 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1"} err="failed to get container status \"04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1\": rpc error: code = NotFound desc = could not find container \"04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1\": container with ID starting with 04e1c545cfc81c618c5b2bf84d92b5ce3f66bd41c7a22e78ac540c3b184343e1 not found: ID does not exist" Sep 15 21:08:49.535124 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.534885 2580 scope.go:117] "RemoveContainer" containerID="49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6" Sep 15 21:08:49.535194 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:08:49.535172 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6\": container with ID starting with 49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6 not found: ID does not exist" containerID="49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6" Sep 15 21:08:49.535258 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.535203 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6"} err="failed to get container status \"49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6\": rpc error: code = NotFound desc = could not find container \"49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6\": container with ID starting with 49d45a63b7000e81ab620e962812859d118216204e6c5e18800d5d6bd214fed6 not found: ID does not exist" Sep 15 21:08:49.535822 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.535801 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de4215-openshift-default-6577d888cb-26n75"] Sep 15 21:08:49.817595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:49.817552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:51.320174 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:51.320141 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" path="/var/lib/kubelet/pods/0817b405-ca0c-4ec7-bdd0-71dc6337f7ce/volumes" Sep 15 21:08:54.565545 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.565510 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j"] Sep 15 21:08:54.566002 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.565767 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" podUID="0793f89d-e418-41d8-94ec-3d7b34ed15b8" containerName="maas-api" containerID="cri-o://1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3" gracePeriod=30 Sep 15 21:08:54.595116 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.594985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:54.692167 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.692122 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:54.795061 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.795032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:54.840023 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.839997 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:08:54.894715 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.894657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:54.928855 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.928818 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793f89d-e418-41d8-94ec-3d7b34ed15b8-openshift-service-ca\") pod \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " Sep 15 21:08:54.929045 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.928865 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-metrics-certs\") pod \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " Sep 15 21:08:54.929045 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.928963 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls\") pod \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " Sep 15 21:08:54.929045 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.928988 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4brs2\" (UniqueName: \"kubernetes.io/projected/0793f89d-e418-41d8-94ec-3d7b34ed15b8-kube-api-access-4brs2\") pod \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\" (UID: \"0793f89d-e418-41d8-94ec-3d7b34ed15b8\") " Sep 15 21:08:54.929466 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.929382 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0793f89d-e418-41d8-94ec-3d7b34ed15b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0793f89d-e418-41d8-94ec-3d7b34ed15b8" (UID: "0793f89d-e418-41d8-94ec-3d7b34ed15b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:08:54.931337 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.931276 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0793f89d-e418-41d8-94ec-3d7b34ed15b8-kube-api-access-4brs2" (OuterVolumeSpecName: "kube-api-access-4brs2") pod "0793f89d-e418-41d8-94ec-3d7b34ed15b8" (UID: "0793f89d-e418-41d8-94ec-3d7b34ed15b8"). InnerVolumeSpecName "kube-api-access-4brs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:08:54.931640 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.931619 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0793f89d-e418-41d8-94ec-3d7b34ed15b8" (UID: "0793f89d-e418-41d8-94ec-3d7b34ed15b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:54.931714 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:54.931665 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0793f89d-e418-41d8-94ec-3d7b34ed15b8" (UID: "0793f89d-e418-41d8-94ec-3d7b34ed15b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:08:55.030797 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.030761 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:55.030797 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.030792 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4brs2\" (UniqueName: \"kubernetes.io/projected/0793f89d-e418-41d8-94ec-3d7b34ed15b8-kube-api-access-4brs2\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:55.030797 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.030802 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793f89d-e418-41d8-94ec-3d7b34ed15b8-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:55.030797 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.030811 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0793f89d-e418-41d8-94ec-3d7b34ed15b8-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:08:55.256174 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.256137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:55.357221 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.357192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:55.552884 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.552801 2580 generic.go:356] "Generic (PLEG): container finished" podID="0793f89d-e418-41d8-94ec-3d7b34ed15b8" containerID="1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3" exitCode=0 Sep 15 21:08:55.552884 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.552846 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" Sep 15 21:08:55.553061 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.552892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" event={"ID":"0793f89d-e418-41d8-94ec-3d7b34ed15b8","Type":"ContainerDied","Data":"1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3"} Sep 15 21:08:55.553061 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.552958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j" event={"ID":"0793f89d-e418-41d8-94ec-3d7b34ed15b8","Type":"ContainerDied","Data":"38742bf2dfc001271a6b3f768534053dd1b3d50c84324ef05b641bc1142bdb97"} Sep 15 21:08:55.553061 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.552992 2580 scope.go:117] "RemoveContainer" containerID="1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3" Sep 15 21:08:55.563016 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.563001 2580 scope.go:117] "RemoveContainer" containerID="1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3" Sep 15 21:08:55.563291 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:08:55.563271 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3\": container with ID starting with 1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3 not found: ID does not exist" containerID="1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3" Sep 15 21:08:55.563338 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.563298 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3"} err="failed to get container status \"1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3\": rpc error: code = NotFound desc = could not find container \"1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3\": container with ID starting with 1832adfd363adae00dd787d2f5c16738947b568ab3559112cd6399b9fda345b3 not found: ID does not exist" Sep 15 21:08:55.575012 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.574990 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j"] Sep 15 21:08:55.585109 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:55.585083 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3825fc-68b5d967b9-rcl2j"] Sep 15 21:08:56.164165 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:56.164127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:56.872305 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:56.872270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:08:57.314530 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:57.314492 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0793f89d-e418-41d8-94ec-3d7b34ed15b8" path="/var/lib/kubelet/pods/0793f89d-e418-41d8-94ec-3d7b34ed15b8/volumes" Sep 15 21:08:57.627622 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:08:57.627507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:00.998620 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:00.998586 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:01.095921 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:01.095886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:01.304296 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:01.304260 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf"] Sep 15 21:09:01.304518 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:01.304491 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" containerID="cri-o://bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98" gracePeriod=30 Sep 15 21:09:01.328742 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:01.328704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:02.327776 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:02.327737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:02.433030 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:02.432989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:03.647924 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:03.647889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:05.748353 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:05.748311 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 21:09:05.748751 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:05.748377 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 21:09:06.558224 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.558199 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:09:06.621373 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.621288 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b997461-188d-4750-b399-d93bc7a537ed" containerID="bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98" exitCode=0 Sep 15 21:09:06.621373 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.621351 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" Sep 15 21:09:06.621585 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.621385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" event={"ID":"7b997461-188d-4750-b399-d93bc7a537ed","Type":"ContainerDied","Data":"bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98"} Sep 15 21:09:06.621585 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.621471 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf" event={"ID":"7b997461-188d-4750-b399-d93bc7a537ed","Type":"ContainerDied","Data":"aac2e75dc503fda0a45a6a37c35b557f25e98e06576e7e98efd7f5b89d30199f"} Sep 15 21:09:06.621585 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.621509 2580 scope.go:117] "RemoveContainer" containerID="bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98" Sep 15 21:09:06.630940 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.630922 2580 scope.go:117] "RemoveContainer" containerID="aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40" Sep 15 21:09:06.639487 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.639470 2580 scope.go:117] "RemoveContainer" containerID="bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98" Sep 15 21:09:06.639772 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:09:06.639753 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98\": container with ID starting with bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98 not found: ID does not exist" containerID="bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98" Sep 15 21:09:06.639825 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.639781 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98"} err="failed to get container status \"bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98\": rpc error: code = NotFound desc = could not find container \"bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98\": container with ID starting with bd4063f194cd5c9993b55ee63ecde723254147434dea18cae7eb5604c27f9f98 not found: ID does not exist" Sep 15 21:09:06.639825 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.639798 2580 scope.go:117] "RemoveContainer" containerID="aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40" Sep 15 21:09:06.640051 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:09:06.640029 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40\": container with ID starting with aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40 not found: ID does not exist" containerID="aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40" Sep 15 21:09:06.640096 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.640058 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40"} err="failed to get container status \"aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40\": rpc error: code = NotFound desc = could not find container \"aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40\": container with ID starting with aa273f4acc10f7945eb7b4f50fedaf8427dfdddb8e96815b384ab0d49060ad40 not found: ID does not exist" Sep 15 21:09:06.649361 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649339 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-certs\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649480 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649373 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-credential-socket\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649480 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649473 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-socket\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649598 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649499 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b997461-188d-4750-b399-d93bc7a537ed-istio-podinfo\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649598 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-envoy\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649598 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649553 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-data\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649754 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649601 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:06.649754 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649624 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b997461-188d-4750-b399-d93bc7a537ed-istiod-ca-cert\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649754 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649694 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-istio-token\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649754 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649699 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:06.649958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649795 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qljdf\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-kube-api-access-qljdf\") pod \"7b997461-188d-4750-b399-d93bc7a537ed\" (UID: \"7b997461-188d-4750-b399-d93bc7a537ed\") " Sep 15 21:09:06.649958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649812 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:06.649958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.649837 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:06.650130 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.650046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b997461-188d-4750-b399-d93bc7a537ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:09:06.650205 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.650187 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.650263 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.650209 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.650263 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.650223 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.650263 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.650238 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.650263 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.650251 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b997461-188d-4750-b399-d93bc7a537ed-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.651694 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.651666 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:06.651803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.651781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:06.651857 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.651845 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-kube-api-access-qljdf" (OuterVolumeSpecName: "kube-api-access-qljdf") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "kube-api-access-qljdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:06.651935 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.651850 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b997461-188d-4750-b399-d93bc7a537ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b997461-188d-4750-b399-d93bc7a537ed" (UID: "7b997461-188d-4750-b399-d93bc7a537ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:09:06.751522 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.751496 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.751522 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.751518 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qljdf\" (UniqueName: \"kubernetes.io/projected/7b997461-188d-4750-b399-d93bc7a537ed-kube-api-access-qljdf\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.751522 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.751527 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b997461-188d-4750-b399-d93bc7a537ed-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.751906 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.751535 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b997461-188d-4750-b399-d93bc7a537ed-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:06.944327 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.944296 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf"] Sep 15 21:09:06.948568 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:06.948541 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3825fc-openshift-default-99bf8554-7nczf"] Sep 15 21:09:07.313764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:07.313726 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b997461-188d-4750-b399-d93bc7a537ed" path="/var/lib/kubelet/pods/7b997461-188d-4750-b399-d93bc7a537ed/volumes" Sep 15 21:09:14.765390 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:14.765351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:25.134032 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.133990 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7755-80b687"] Sep 15 21:09:25.134684 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134660 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134686 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134700 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134708 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134728 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0793f89d-e418-41d8-94ec-3d7b34ed15b8" containerName="maas-api" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134739 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0793f89d-e418-41d8-94ec-3d7b34ed15b8" containerName="maas-api" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134759 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134767 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134777 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" containerName="main" Sep 15 21:09:25.134782 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134785 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" containerName="main" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134799 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1af04f-1277-4565-8f0b-344bce74c765" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134807 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1af04f-1277-4565-8f0b-344bce74c765" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134837 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134846 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134856 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3d2622-0cac-4f1a-86fa-20a6613e4811" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134864 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3d2622-0cac-4f1a-86fa-20a6613e4811" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134878 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0180a0cf-01c3-4c36-8f7a-485107ebd453" containerName="payload-pre-processing" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134888 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0180a0cf-01c3-4c36-8f7a-485107ebd453" containerName="payload-pre-processing" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134900 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.134908 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135008 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135022 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9bcad3-3d3c-46c9-a855-1b739d78cf8c" containerName="main" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135033 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135042 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135052 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135062 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b4b1ec-f9af-4b73-a2de-60b17235156f" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135072 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135082 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0180a0cf-01c3-4c36-8f7a-485107ebd453" containerName="payload-pre-processing" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135091 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d818ed2-4ec8-4cf6-a46a-d36bcd21ef48" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135101 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0793f89d-e418-41d8-94ec-3d7b34ed15b8" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135110 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3d2622-0cac-4f1a-86fa-20a6613e4811" containerName="maas-api" Sep 15 21:09:25.135236 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.135119 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1af04f-1277-4565-8f0b-344bce74c765" containerName="maas-api" Sep 15 21:09:25.140215 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.140189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:25.142465 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.142439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cs2t2\"" Sep 15 21:09:25.143544 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.143517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-80b687"] Sep 15 21:09:25.216956 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.216921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/32bdaf9f-a03a-4363-8fe9-d2767005ebb2-kube-api-access-xsm5q\") pod \"test-curl-7755-80b687\" (UID: \"32bdaf9f-a03a-4363-8fe9-d2767005ebb2\") " pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:25.317401 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.317370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/32bdaf9f-a03a-4363-8fe9-d2767005ebb2-kube-api-access-xsm5q\") pod \"test-curl-7755-80b687\" (UID: \"32bdaf9f-a03a-4363-8fe9-d2767005ebb2\") " pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:25.325691 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.325665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/32bdaf9f-a03a-4363-8fe9-d2767005ebb2-kube-api-access-xsm5q\") pod \"test-curl-7755-80b687\" (UID: \"32bdaf9f-a03a-4363-8fe9-d2767005ebb2\") " pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:25.452071 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.451976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:25.725735 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.725706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-80b687"] Sep 15 21:09:25.727774 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:09:25.727742 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32bdaf9f_a03a_4363_8fe9_d2767005ebb2.slice/crio-feffdca3efbebdfff2e13c9bcf911a56442f9937d4f66be8a4c1087ce9c5aeb0 WatchSource:0}: Error finding container feffdca3efbebdfff2e13c9bcf911a56442f9937d4f66be8a4c1087ce9c5aeb0: Status 404 returned error can't find the container with id feffdca3efbebdfff2e13c9bcf911a56442f9937d4f66be8a4c1087ce9c5aeb0 Sep 15 21:09:25.747547 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:25.747464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-80b687" event={"ID":"32bdaf9f-a03a-4363-8fe9-d2767005ebb2","Type":"ContainerStarted","Data":"feffdca3efbebdfff2e13c9bcf911a56442f9937d4f66be8a4c1087ce9c5aeb0"} Sep 15 21:09:26.758857 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:26.758815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-80b687" event={"ID":"32bdaf9f-a03a-4363-8fe9-d2767005ebb2","Type":"ContainerStarted","Data":"72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823"} Sep 15 21:09:26.774969 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:26.774924 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7755-80b687" podStartSLOduration=1.7749111640000002 podStartE2EDuration="1.774911164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:26.771677058 +0000 UTC m=+1955.901413643" watchObservedRunningTime="2026-09-15 21:09:26.774911164 +0000 UTC m=+1955.904647747" Sep 15 21:09:27.413473 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:27.413439 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7755-80b687"] Sep 15 21:09:27.418768 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:27.418737 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7755-80b687"] Sep 15 21:09:27.765193 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:09:27.765159 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7755-80b687" container="test-curl-7755-80b687" Sep 15 21:09:28.770954 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:28.770901 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7755-80b687" podUID="32bdaf9f-a03a-4363-8fe9-d2767005ebb2" containerName="test-curl-7755-80b687" containerID="cri-o://72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823" gracePeriod=2 Sep 15 21:09:28.772952 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:09:28.772918 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-80b687\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="32bdaf9f-a03a-4363-8fe9-d2767005ebb2" pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:28.915240 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:28.915205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:29.669005 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:29.668970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:30.282553 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:30.282521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:30.766599 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:30.766566 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:30.947415 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:30.947378 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:30.969530 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:30.969502 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/32bdaf9f-a03a-4363-8fe9-d2767005ebb2-kube-api-access-xsm5q\") pod \"32bdaf9f-a03a-4363-8fe9-d2767005ebb2\" (UID: \"32bdaf9f-a03a-4363-8fe9-d2767005ebb2\") " Sep 15 21:09:30.971457 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:30.971433 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bdaf9f-a03a-4363-8fe9-d2767005ebb2-kube-api-access-xsm5q" (OuterVolumeSpecName: "kube-api-access-xsm5q") pod "32bdaf9f-a03a-4363-8fe9-d2767005ebb2" (UID: "32bdaf9f-a03a-4363-8fe9-d2767005ebb2"). InnerVolumeSpecName "kube-api-access-xsm5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:31.070637 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.070571 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/32bdaf9f-a03a-4363-8fe9-d2767005ebb2-kube-api-access-xsm5q\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:31.315056 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.315004 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32bdaf9f-a03a-4363-8fe9-d2767005ebb2" path="/var/lib/kubelet/pods/32bdaf9f-a03a-4363-8fe9-d2767005ebb2/volumes" Sep 15 21:09:31.598056 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.597957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:31.791818 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.791779 2580 generic.go:356] "Generic (PLEG): container finished" podID="32bdaf9f-a03a-4363-8fe9-d2767005ebb2" containerID="72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823" exitCode=137 Sep 15 21:09:31.792011 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.791842 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80b687" Sep 15 21:09:31.792011 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.791894 2580 scope.go:117] "RemoveContainer" containerID="72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823" Sep 15 21:09:31.802500 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.802484 2580 scope.go:117] "RemoveContainer" containerID="72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823" Sep 15 21:09:31.802805 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:09:31.802774 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823\": container with ID starting with 72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823 not found: ID does not exist" containerID="72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823" Sep 15 21:09:31.802905 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:31.802812 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823"} err="failed to get container status \"72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823\": rpc error: code = NotFound desc = could not find container \"72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823\": container with ID starting with 72dde1e030abdb587212cb6e62b1f7aebea3d137da34e7cace10adcacda5a823 not found: ID does not exist" Sep 15 21:09:32.529817 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:32.529774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:33.297098 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:33.297066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:34.736735 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:34.736698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:42.380344 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:42.380304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:45.397547 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:45.397454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:45.399499 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:45.399475 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv"] Sep 15 21:09:45.401890 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:45.401868 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-16bde4-29825100-2t2cv"] Sep 15 21:09:45.465453 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:45.465385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:45.564123 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:45.564091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:45.667524 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:45.667395 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:45.997364 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:45.997266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:46.562397 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:46.562348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:46.826343 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:46.826249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:47.314623 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:47.314579 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbf6735-09ce-403a-a7c6-2f35572fb7eb" path="/var/lib/kubelet/pods/6bbf6735-09ce-403a-a7c6-2f35572fb7eb/volumes" Sep 15 21:09:47.813074 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:47.813023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:51.620924 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:51.620894 2580 scope.go:117] "RemoveContainer" containerID="e2b74cc8d317513a415fb8b59c6f7dde12132d293b0432e8ea05716232733805" Sep 15 21:09:52.280819 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:52.280778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:52.382265 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:52.382221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:52.531645 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:52.531560 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5"] Sep 15 21:09:52.531843 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:52.531817 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" containerID="cri-o://d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74" gracePeriod=30 Sep 15 21:09:52.554963 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:52.554928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:09:57.782087 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.782063 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 21:09:57.934892 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.934812 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-credential-socket\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.934892 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.934850 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc431a76-246a-4136-a3eb-a594eba56ed3-istiod-ca-cert\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.934892 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.934866 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-envoy\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.935152 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.934908 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-podinfo\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.935152 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935021 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-certs\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.935152 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935100 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-socket\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.935152 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935109 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:57.935152 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935128 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64ckt\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-kube-api-access-64ckt\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.935152 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935152 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-data\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.935482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935196 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-token\") pod \"cc431a76-246a-4136-a3eb-a594eba56ed3\" (UID: \"cc431a76-246a-4136-a3eb-a594eba56ed3\") " Sep 15 21:09:57.935482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935313 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:57.935482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc431a76-246a-4136-a3eb-a594eba56ed3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:09:57.935482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935345 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:57.935689 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935545 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:57.935689 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935660 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:57.935689 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935683 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc431a76-246a-4136-a3eb-a594eba56ed3-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:57.935858 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935698 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:57.935858 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935711 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:57.935858 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.935725 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:57.937185 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.937152 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:09:57.937185 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.937166 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:09:57.937359 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.937272 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:57.937400 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.937358 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-kube-api-access-64ckt" (OuterVolumeSpecName: "kube-api-access-64ckt") pod "cc431a76-246a-4136-a3eb-a594eba56ed3" (UID: "cc431a76-246a-4136-a3eb-a594eba56ed3"). InnerVolumeSpecName "kube-api-access-64ckt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:09:57.952029 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.952003 2580 generic.go:356] "Generic (PLEG): container finished" podID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerID="d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74" exitCode=0 Sep 15 21:09:57.952148 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.952056 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" Sep 15 21:09:57.952148 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.952078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" event={"ID":"cc431a76-246a-4136-a3eb-a594eba56ed3","Type":"ContainerDied","Data":"d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74"} Sep 15 21:09:57.952148 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.952132 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5" event={"ID":"cc431a76-246a-4136-a3eb-a594eba56ed3","Type":"ContainerDied","Data":"5fbbdfee4bdb00f578d7a1c2911a2a163bd52a5adbf2a344c16ef99173f6a271"} Sep 15 21:09:57.952301 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.952155 2580 scope.go:117] "RemoveContainer" containerID="d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74" Sep 15 21:09:57.962583 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.962547 2580 scope.go:117] "RemoveContainer" containerID="6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f" Sep 15 21:09:57.971176 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.971160 2580 scope.go:117] "RemoveContainer" containerID="d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74" Sep 15 21:09:57.971451 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:09:57.971432 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74\": container with ID starting with d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74 not found: ID does not exist" containerID="d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74" Sep 15 21:09:57.971520 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.971459 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74"} err="failed to get container status \"d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74\": rpc error: code = NotFound desc = could not find container \"d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74\": container with ID starting with d459785cdb9ffa4183818b78b531eb6d5fca67c9624582cf3d4b92df24e2af74 not found: ID does not exist" Sep 15 21:09:57.971520 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.971475 2580 scope.go:117] "RemoveContainer" containerID="6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f" Sep 15 21:09:57.971721 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:09:57.971705 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f\": container with ID starting with 6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f not found: ID does not exist" containerID="6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f" Sep 15 21:09:57.971759 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:57.971725 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f"} err="failed to get container status \"6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f\": rpc error: code = NotFound desc = could not find container \"6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f\": container with ID starting with 6a87c6934efa7d2d888e50f72266875ae8ad31de4f28651fc4df9f274b30613f not found: ID does not exist" Sep 15 21:09:58.022440 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:58.022392 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5"] Sep 15 21:09:58.025527 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:58.025502 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-16bde4-openshift-default-74f788db76-4q8n5"] Sep 15 21:09:58.036622 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:58.036601 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:58.036694 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:58.036640 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ckt\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-kube-api-access-64ckt\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:58.036694 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:58.036651 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:58.036694 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:58.036659 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc431a76-246a-4136-a3eb-a594eba56ed3-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:09:59.315557 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:09:59.315520 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" path="/var/lib/kubelet/pods/cc431a76-246a-4136-a3eb-a594eba56ed3/volumes" Sep 15 21:10:11.582599 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:11.582563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:10:11.682798 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:11.682760 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:10:11.961745 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:11.961658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:10:15.898751 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:15.898706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:10:26.510812 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:26.510774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:10:44.273139 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273107 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7755-ffc702"] Sep 15 21:10:44.273581 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273561 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" Sep 15 21:10:44.273581 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273576 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273584 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273590 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273599 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32bdaf9f-a03a-4363-8fe9-d2767005ebb2" containerName="test-curl-7755-80b687" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273606 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32bdaf9f-a03a-4363-8fe9-d2767005ebb2" containerName="test-curl-7755-80b687" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273615 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273620 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b997461-188d-4750-b399-d93bc7a537ed" containerName="istio-proxy" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273627 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" Sep 15 21:10:44.273660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273632 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0817b405-ca0c-4ec7-bdd0-71dc6337f7ce" containerName="istio-proxy" Sep 15 21:10:44.273932 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273694 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" Sep 15 21:10:44.273932 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273703 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc431a76-246a-4136-a3eb-a594eba56ed3" containerName="istio-proxy" Sep 15 21:10:44.273932 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.273712 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32bdaf9f-a03a-4363-8fe9-d2767005ebb2" containerName="test-curl-7755-80b687" Sep 15 21:10:44.276805 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.276789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:44.279387 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.279360 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cs2t2\"" Sep 15 21:10:44.289817 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.282865 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-ffc702"] Sep 15 21:10:44.385743 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.385708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kp4l\" (UniqueName: \"kubernetes.io/projected/be11d6bc-0f79-435f-900e-580097217abe-kube-api-access-8kp4l\") pod \"test-curl-7755-ffc702\" (UID: \"be11d6bc-0f79-435f-900e-580097217abe\") " pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:44.486880 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.486842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kp4l\" (UniqueName: \"kubernetes.io/projected/be11d6bc-0f79-435f-900e-580097217abe-kube-api-access-8kp4l\") pod \"test-curl-7755-ffc702\" (UID: \"be11d6bc-0f79-435f-900e-580097217abe\") " pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:44.494672 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.494636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kp4l\" (UniqueName: \"kubernetes.io/projected/be11d6bc-0f79-435f-900e-580097217abe-kube-api-access-8kp4l\") pod \"test-curl-7755-ffc702\" (UID: \"be11d6bc-0f79-435f-900e-580097217abe\") " pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:44.596198 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.596128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:44.870268 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:44.870190 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-ffc702"] Sep 15 21:10:45.234887 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:45.234805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-ffc702" event={"ID":"be11d6bc-0f79-435f-900e-580097217abe","Type":"ContainerStarted","Data":"57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a"} Sep 15 21:10:45.234887 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:45.234851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-ffc702" event={"ID":"be11d6bc-0f79-435f-900e-580097217abe","Type":"ContainerStarted","Data":"b513e8487d50dbb54222f113a1ff4aad5256c87fa71aecdfdc54ea10b5e8d147"} Sep 15 21:10:45.250556 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:45.250507 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7755-ffc702" podStartSLOduration=1.25049232 podStartE2EDuration="1.25049232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:45.248186589 +0000 UTC m=+2034.377923172" watchObservedRunningTime="2026-09-15 21:10:45.25049232 +0000 UTC m=+2034.380228902" Sep 15 21:10:45.881856 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:45.881809 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7755-ffc702"] Sep 15 21:10:45.885960 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:45.885932 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7755-ffc702"] Sep 15 21:10:46.012776 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.012734 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7755-2e8a6a"] Sep 15 21:10:46.013203 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.013186 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be11d6bc-0f79-435f-900e-580097217abe" containerName="test-curl-7755-ffc702" Sep 15 21:10:46.013279 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.013205 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be11d6bc-0f79-435f-900e-580097217abe" containerName="test-curl-7755-ffc702" Sep 15 21:10:46.013332 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.013311 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be11d6bc-0f79-435f-900e-580097217abe" containerName="test-curl-7755-ffc702" Sep 15 21:10:46.015972 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.015951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:46.022893 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.022862 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-2e8a6a"] Sep 15 21:10:46.204146 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.204115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmqj\" (UniqueName: \"kubernetes.io/projected/6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce-kube-api-access-kwmqj\") pod \"test-curl-7755-2e8a6a\" (UID: \"6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce\") " pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:46.240765 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:46.240731 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7755-ffc702" container="test-curl-7755-ffc702" Sep 15 21:10:46.304856 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.304766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmqj\" (UniqueName: \"kubernetes.io/projected/6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce-kube-api-access-kwmqj\") pod \"test-curl-7755-2e8a6a\" (UID: \"6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce\") " pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:46.312641 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.312613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmqj\" (UniqueName: \"kubernetes.io/projected/6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce-kube-api-access-kwmqj\") pod \"test-curl-7755-2e8a6a\" (UID: \"6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce\") " pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:46.327186 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.327160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:46.597629 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:46.597599 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-2e8a6a"] Sep 15 21:10:46.599321 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:10:46.599293 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dc6de7d_dbb5_4bcd_b5cd_6e73411df2ce.slice/crio-47b3883c40ac69b3bb57341cb8e51cf569ceeadb2e5007b9f55206bfeb0a5267 WatchSource:0}: Error finding container 47b3883c40ac69b3bb57341cb8e51cf569ceeadb2e5007b9f55206bfeb0a5267: Status 404 returned error can't find the container with id 47b3883c40ac69b3bb57341cb8e51cf569ceeadb2e5007b9f55206bfeb0a5267 Sep 15 21:10:47.248161 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:47.248124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-2e8a6a" event={"ID":"6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce","Type":"ContainerStarted","Data":"dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef"} Sep 15 21:10:47.248712 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:47.248173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-2e8a6a" event={"ID":"6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce","Type":"ContainerStarted","Data":"47b3883c40ac69b3bb57341cb8e51cf569ceeadb2e5007b9f55206bfeb0a5267"} Sep 15 21:10:47.248712 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:47.248248 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7755-ffc702" podUID="be11d6bc-0f79-435f-900e-580097217abe" containerName="test-curl-7755-ffc702" containerID="cri-o://57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a" gracePeriod=2 Sep 15 21:10:47.250246 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:47.250220 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-ffc702\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="be11d6bc-0f79-435f-900e-580097217abe" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:47.266530 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:47.266502 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-ffc702\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="be11d6bc-0f79-435f-900e-580097217abe" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:47.267392 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:47.267353 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7755-2e8a6a" podStartSLOduration=2.267338894 podStartE2EDuration="2.267338894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:47.264690973 +0000 UTC m=+2036.394427557" watchObservedRunningTime="2026-09-15 21:10:47.267338894 +0000 UTC m=+2036.397075476" Sep 15 21:10:47.910521 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:47.910486 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7755-2e8a6a"] Sep 15 21:10:47.914565 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:47.914534 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7755-2e8a6a"] Sep 15 21:10:48.255461 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:48.255425 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7755-2e8a6a" container="test-curl-7755-2e8a6a" Sep 15 21:10:49.260972 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:49.260939 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7755-2e8a6a" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" containerName="test-curl-7755-2e8a6a" containerID="cri-o://dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef" gracePeriod=2 Sep 15 21:10:49.263135 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:49.263107 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-2e8a6a\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:49.427073 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:49.427048 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:49.430347 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:49.430325 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kp4l\" (UniqueName: \"kubernetes.io/projected/be11d6bc-0f79-435f-900e-580097217abe-kube-api-access-8kp4l\") pod \"be11d6bc-0f79-435f-900e-580097217abe\" (UID: \"be11d6bc-0f79-435f-900e-580097217abe\") " Sep 15 21:10:49.432297 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:49.432273 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be11d6bc-0f79-435f-900e-580097217abe-kube-api-access-8kp4l" (OuterVolumeSpecName: "kube-api-access-8kp4l") pod "be11d6bc-0f79-435f-900e-580097217abe" (UID: "be11d6bc-0f79-435f-900e-580097217abe"). InnerVolumeSpecName "kube-api-access-8kp4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:10:49.531333 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:49.531260 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kp4l\" (UniqueName: \"kubernetes.io/projected/be11d6bc-0f79-435f-900e-580097217abe-kube-api-access-8kp4l\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:10:50.092301 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:50.092251 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:10:50.268057 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:50.268019 2580 generic.go:356] "Generic (PLEG): container finished" podID="be11d6bc-0f79-435f-900e-580097217abe" containerID="57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a" exitCode=137 Sep 15 21:10:50.268548 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:50.268060 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:50.268548 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:50.268123 2580 scope.go:117] "RemoveContainer" containerID="57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a" Sep 15 21:10:50.278807 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:50.278774 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-ffc702\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="be11d6bc-0f79-435f-900e-580097217abe" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:50.278891 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:50.278825 2580 scope.go:117] "RemoveContainer" containerID="57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a" Sep 15 21:10:50.279125 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:50.279106 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a\": container with ID starting with 57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a not found: ID does not exist" containerID="57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a" Sep 15 21:10:50.279168 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:50.279135 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a"} err="failed to get container status \"57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a\": rpc error: code = NotFound desc = could not find container \"57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a\": container with ID starting with 57a3488901d77dcbdcbcf22020c4bcacca49fe15c5f3094e6f5c72e388975f4a not found: ID does not exist" Sep 15 21:10:51.196221 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:51.196188 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:10:51.313586 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:51.313551 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-ffc702\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="be11d6bc-0f79-435f-900e-580097217abe" pod="opendatahub/test-curl-7755-ffc702" Sep 15 21:10:51.316244 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:51.316215 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be11d6bc-0f79-435f-900e-580097217abe" path="/var/lib/kubelet/pods/be11d6bc-0f79-435f-900e-580097217abe/volumes" Sep 15 21:10:51.455766 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:51.455705 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:51.457737 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:51.457706 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-2e8a6a\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:51.552476 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:51.552442 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwmqj\" (UniqueName: \"kubernetes.io/projected/6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce-kube-api-access-kwmqj\") pod \"6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce\" (UID: \"6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce\") " Sep 15 21:10:51.554471 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:51.554441 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce-kube-api-access-kwmqj" (OuterVolumeSpecName: "kube-api-access-kwmqj") pod "6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" (UID: "6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce"). InnerVolumeSpecName "kube-api-access-kwmqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:10:51.653321 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:51.653286 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmqj\" (UniqueName: \"kubernetes.io/projected/6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce-kube-api-access-kwmqj\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:10:52.285943 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:52.285904 2580 generic.go:356] "Generic (PLEG): container finished" podID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" containerID="dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef" exitCode=137 Sep 15 21:10:52.286122 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:52.285947 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:52.286122 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:52.285966 2580 scope.go:117] "RemoveContainer" containerID="dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef" Sep 15 21:10:52.287955 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:52.287931 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-2e8a6a\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:52.296099 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:52.296080 2580 scope.go:117] "RemoveContainer" containerID="dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef" Sep 15 21:10:52.296372 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:52.296352 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef\": container with ID starting with dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef not found: ID does not exist" containerID="dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef" Sep 15 21:10:52.296473 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:52.296381 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef"} err="failed to get container status \"dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef\": rpc error: code = NotFound desc = could not find container \"dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef\": container with ID starting with dc3f861bf043e809a026e212ed49548b69248b3cd92c52403bbb3602b651d4ef not found: ID does not exist" Sep 15 21:10:52.296855 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:10:52.296839 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-2e8a6a\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" pod="opendatahub/test-curl-7755-2e8a6a" Sep 15 21:10:53.314605 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:10:53.314564 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" path="/var/lib/kubelet/pods/6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce/volumes" Sep 15 21:11:11.192812 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:11:11.192770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:11:12.817350 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:11:12.817316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:11:26.400202 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:11:26.400113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:11:31.745305 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:11:31.745270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:11:47.985337 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:11:47.985298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:11:51.513133 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:11:51.513100 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:11:51.520523 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:11:51.520500 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:12:03.917860 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:03.917817 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:04.903272 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:04.903239 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:05.359301 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.359269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:05.628669 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.628587 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk"] Sep 15 21:12:05.629250 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.629231 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" containerName="test-curl-7755-2e8a6a" Sep 15 21:12:05.629312 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.629254 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" containerName="test-curl-7755-2e8a6a" Sep 15 21:12:05.629397 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.629384 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc6de7d-dbb5-4bcd-b5cd-6e73411df2ce" containerName="test-curl-7755-2e8a6a" Sep 15 21:12:05.632492 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.632470 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.638375 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.638355 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-219de8-openshift-default-dockercfg-8t569\"" Sep 15 21:12:05.654569 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.654542 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk"] Sep 15 21:12:05.677080 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.677052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:05.750613 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b331c713-9273-4db3-ae12-43159ab241b5-istio-podinfo\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.750803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-envoy\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.750803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-data\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.750803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b331c713-9273-4db3-ae12-43159ab241b5-istiod-ca-cert\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.750803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-credential-socket\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.751010 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-certs\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.751010 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-socket\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.751010 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbrw\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-kube-api-access-cgbrw\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.751010 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.750985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-istio-token\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852032 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.851994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-istio-token\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b331c713-9273-4db3-ae12-43159ab241b5-istio-podinfo\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-envoy\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-data\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b331c713-9273-4db3-ae12-43159ab241b5-istiod-ca-cert\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-credential-socket\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-certs\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-socket\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852652 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbrw\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-kube-api-access-cgbrw\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852721 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852678 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-certs\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852721 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-data\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852838 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852741 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-credential-socket\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.852838 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.852796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-socket\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.853337 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.853312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b331c713-9273-4db3-ae12-43159ab241b5-istiod-ca-cert\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.854487 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.854469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-envoy\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.854590 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.854535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b331c713-9273-4db3-ae12-43159ab241b5-istio-podinfo\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.863713 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.863691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbrw\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-kube-api-access-cgbrw\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.864544 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.864525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-istio-token\") pod \"e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.943848 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.943774 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:05.971294 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:05.971257 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:06.225088 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.225059 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk"] Sep 15 21:12:06.226722 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:12:06.226694 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb331c713_9273_4db3_ae12_43159ab241b5.slice/crio-dc547a23d30e0c41889354414221b89161ce3f046cb403b6dbf051b36cb48409 WatchSource:0}: Error finding container dc547a23d30e0c41889354414221b89161ce3f046cb403b6dbf051b36cb48409: Status 404 returned error can't find the container with id dc547a23d30e0c41889354414221b89161ce3f046cb403b6dbf051b36cb48409 Sep 15 21:12:06.229012 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.228981 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:12:06.229101 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.229042 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:12:06.229101 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.229071 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:12:06.739441 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.739394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" event={"ID":"b331c713-9273-4db3-ae12-43159ab241b5","Type":"ContainerStarted","Data":"2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a"} Sep 15 21:12:06.739798 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.739457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" event={"ID":"b331c713-9273-4db3-ae12-43159ab241b5","Type":"ContainerStarted","Data":"dc547a23d30e0c41889354414221b89161ce3f046cb403b6dbf051b36cb48409"} Sep 15 21:12:06.760885 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.760833 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podStartSLOduration=1.760817684 podStartE2EDuration="1.760817684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:06.757561586 +0000 UTC m=+2115.887298169" watchObservedRunningTime="2026-09-15 21:12:06.760817684 +0000 UTC m=+2115.890554267" Sep 15 21:12:06.944162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:06.944130 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:07.949993 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:07.949958 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:08.473654 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:08.473620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:08.706461 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:08.706426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:08.757909 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:08.757878 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:08.758787 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:08.758764 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:08.829659 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:08.829627 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:09.528718 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:09.528676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:10.234797 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.234745 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842"] Sep 15 21:12:10.238770 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.238744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.241062 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.241037 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-219de8-serving-cert\"" Sep 15 21:12:10.250983 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.250955 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842"] Sep 15 21:12:10.273999 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.273972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:10.373252 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.373214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:10.400832 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.400800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6dj\" (UniqueName: \"kubernetes.io/projected/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-kube-api-access-8d6dj\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.400979 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.400841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.400979 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.400863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-metrics-certs\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.400979 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.400949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-maas-api-tls\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.501903 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.501870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6dj\" (UniqueName: \"kubernetes.io/projected/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-kube-api-access-8d6dj\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.502116 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.501917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.502116 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.501940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-metrics-certs\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.502116 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.502056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-maas-api-tls\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.502705 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.502679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.504450 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.504422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-maas-api-tls\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.504549 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.504522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-metrics-certs\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.512155 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.512132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6dj\" (UniqueName: \"kubernetes.io/projected/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-kube-api-access-8d6dj\") pod \"maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.546043 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.546003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:10.552813 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.552789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:10.748470 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.748399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:10.831778 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.831750 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842"] Sep 15 21:12:10.836974 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.836650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:10.837641 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:10.837617 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:12:11.777928 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:11.777892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" event={"ID":"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36","Type":"ContainerStarted","Data":"c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d"} Sep 15 21:12:11.778310 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:11.777953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" event={"ID":"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36","Type":"ContainerStarted","Data":"3323fa55b82571e4a757405bf1ed1e5a597cf36f96f8c27a6a5d4058930107e8"} Sep 15 21:12:11.778310 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:11.778128 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:11.795477 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:11.795433 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" podStartSLOduration=1.315853856 podStartE2EDuration="1.79541822s" totalImagesPullingTime="479.564364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:11.793998667 +0000 UTC m=+2120.923735251" watchObservedRunningTime="2026-09-15 21:12:11.79541822 +0000 UTC m=+2120.925154798" Sep 15 21:12:12.174988 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:12.174900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:17.789814 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:17.789784 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:12:23.759573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.759536 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 15 21:12:23.760048 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.759602 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 15 21:12:23.863844 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.863763 2580 generic.go:356] "Generic (PLEG): container finished" podID="b331c713-9273-4db3-ae12-43159ab241b5" containerID="2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a" exitCode=0 Sep 15 21:12:23.863995 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.863837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" event={"ID":"b331c713-9273-4db3-ae12-43159ab241b5","Type":"ContainerDied","Data":"2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a"} Sep 15 21:12:23.864260 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.864247 2580 scope.go:117] "RemoveContainer" containerID="2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a" Sep 15 21:12:23.865311 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.865286 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:12:23.865367 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.865348 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:12:23.865416 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:23.865379 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 15 21:12:24.871841 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:24.871799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" event={"ID":"b331c713-9273-4db3-ae12-43159ab241b5","Type":"ContainerStarted","Data":"e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7"} Sep 15 21:12:24.944784 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:24.944744 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:24.945000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:24.944972 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 15 21:12:24.945085 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:24.945038 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 15 21:12:25.944450 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:25.944389 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 15 21:12:25.944852 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:25.944481 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 15 21:12:26.944803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:26.944764 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 15 21:12:26.945282 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:26.944846 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 15 21:12:27.945030 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:27.944990 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 15 21:12:27.945451 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:27.945056 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 15 21:12:28.944670 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:28.944632 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 15 21:12:28.944870 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:28.944705 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 15 21:12:29.945116 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:29.945066 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 15 21:12:29.945566 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:29.945143 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 15 21:12:30.948555 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:30.948523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:30.948933 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:30.948865 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:30.949573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:30.949554 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:12:31.172343 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:31.172308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:32.494945 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:32.494906 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:33.149429 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:33.149384 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:33.683381 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:33.683339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:34.038329 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:34.038292 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:35.788587 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:35.788551 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:37.242829 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:37.242789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:37.922830 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:37.922788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:38.011011 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:38.010971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:38.303706 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:38.303674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:39.124310 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:39.124276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:39.850618 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:39.850576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:40.612371 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.612329 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng"] Sep 15 21:12:40.618154 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.618127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.620337 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.620312 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7f09a7-serving-cert\"" Sep 15 21:12:40.629050 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.629026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng"] Sep 15 21:12:40.645869 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.645837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:40.702399 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.702372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.702548 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.702498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.702605 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.702552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jkl\" (UniqueName: \"kubernetes.io/projected/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-kube-api-access-g7jkl\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.702605 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.702596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.722473 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.720569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:40.803784 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.803750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.803946 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.803806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jkl\" (UniqueName: \"kubernetes.io/projected/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-kube-api-access-g7jkl\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.803946 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.803843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.803946 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.803896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.804083 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:12:40.803997 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-serving-cert: secret "maas-api-e2e-ipp-b-7f09a7-serving-cert" not found Sep 15 21:12:40.804128 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:12:40.804085 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls podName:93b8c55d-cec6-4fc6-ad65-a61c82ed053a nodeName:}" failed. No retries permitted until 2026-09-15 21:12:41.304033031 +0000 UTC m=+2150.433769595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls") pod "maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" (UID: "93b8c55d-cec6-4fc6-ad65-a61c82ed053a") : secret "maas-api-e2e-ipp-b-7f09a7-serving-cert" not found Sep 15 21:12:40.804577 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.804553 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.806234 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.806207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.816207 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.816180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jkl\" (UniqueName: \"kubernetes.io/projected/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-kube-api-access-g7jkl\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:40.996999 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:40.996923 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:41.125201 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:41.125155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:41.215559 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:41.215518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:41.308021 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:41.307989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:41.310514 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:41.310482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:41.531672 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:41.531636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:41.807551 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:41.807521 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng"] Sep 15 21:12:41.808981 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:12:41.808953 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b8c55d_cec6_4fc6_ad65_a61c82ed053a.slice/crio-c025056bd261b14baed35d2f98220e88d6a7c42e8bf128f632197bcfebe2aa61 WatchSource:0}: Error finding container c025056bd261b14baed35d2f98220e88d6a7c42e8bf128f632197bcfebe2aa61: Status 404 returned error can't find the container with id c025056bd261b14baed35d2f98220e88d6a7c42e8bf128f632197bcfebe2aa61 Sep 15 21:12:41.978522 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:41.978434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" event={"ID":"93b8c55d-cec6-4fc6-ad65-a61c82ed053a","Type":"ContainerStarted","Data":"c025056bd261b14baed35d2f98220e88d6a7c42e8bf128f632197bcfebe2aa61"} Sep 15 21:12:42.987427 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:42.987371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" event={"ID":"93b8c55d-cec6-4fc6-ad65-a61c82ed053a","Type":"ContainerStarted","Data":"f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661"} Sep 15 21:12:42.987815 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:42.987455 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:12:43.003312 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:43.003257 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" podStartSLOduration=2.666844633 podStartE2EDuration="3.003240867s" totalImagesPullingTime="336.396234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:43.002775026 +0000 UTC m=+2152.132511612" watchObservedRunningTime="2026-09-15 21:12:43.003240867 +0000 UTC m=+2152.132977449" Sep 15 21:12:48.256175 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:48.256115 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:12:49.002654 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:12:49.002619 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:13:02.433724 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:02.433685 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:03.756511 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:03.756475 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:04.538118 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:04.538072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:05.174857 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:05.174816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:05.827565 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:05.827525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:06.525256 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:06.525225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:08.005952 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:08.005919 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:16.988804 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:16.988761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:44.347193 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:44.347148 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:44.648463 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:44.648314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:44.934374 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:44.934265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:44.989897 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:44.989871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:45.309200 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:45.309160 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:46.536271 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:46.536235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:47.355388 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:47.355351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:47.447459 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:47.447424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:48.128835 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:48.128789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:49.124730 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:49.124691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:49.790086 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:49.790052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:50.416602 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.416566 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7"] Sep 15 21:13:50.422211 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.422184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.424257 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.424231 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a6f49f-serving-cert\"" Sep 15 21:13:50.430563 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.430539 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7"] Sep 15 21:13:50.449510 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.449469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:50.486775 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.486739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.486958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.486846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh6c\" (UniqueName: \"kubernetes.io/projected/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-kube-api-access-sxh6c\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.486958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.486877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.487076 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.486964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.514899 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.514863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:50.588283 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.588247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh6c\" (UniqueName: \"kubernetes.io/projected/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-kube-api-access-sxh6c\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.588501 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.588292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.588501 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.588331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.588501 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.588446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.588501 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:13:50.588479 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-serving-cert: secret "maas-api-e2e-ipp-cleanup-a6f49f-serving-cert" not found Sep 15 21:13:50.588751 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:13:50.588553 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls podName:452f021b-1cd2-4a35-9b1b-89d6a0c533cb nodeName:}" failed. No retries permitted until 2026-09-15 21:13:51.088525394 +0000 UTC m=+2220.218261963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" (UID: "452f021b-1cd2-4a35-9b1b-89d6a0c533cb") : secret "maas-api-e2e-ipp-cleanup-a6f49f-serving-cert" not found Sep 15 21:13:50.589036 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.589015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.590983 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.590961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.598139 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.598116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh6c\" (UniqueName: \"kubernetes.io/projected/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-kube-api-access-sxh6c\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:50.824001 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.823968 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:50.927152 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:50.927120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:51.035186 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:51.035146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:51.095573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:51.095489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:51.098142 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:51.098119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:51.335262 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:51.335233 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:51.554907 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:51.554870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:13:51.612163 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:51.612126 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7"] Sep 15 21:13:51.614548 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:13:51.614519 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452f021b_1cd2_4a35_9b1b_89d6a0c533cb.slice/crio-8392688f8e4e1a6e55760593db20897feb31b74df701547102808e1eb39986fd WatchSource:0}: Error finding container 8392688f8e4e1a6e55760593db20897feb31b74df701547102808e1eb39986fd: Status 404 returned error can't find the container with id 8392688f8e4e1a6e55760593db20897feb31b74df701547102808e1eb39986fd Sep 15 21:13:52.414105 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:52.414005 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" event={"ID":"452f021b-1cd2-4a35-9b1b-89d6a0c533cb","Type":"ContainerStarted","Data":"e474c9966c3286aec3916df3ba61a7200f1df45d3c912915c429d0621449eaf2"} Sep 15 21:13:52.414105 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:52.414081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:13:52.414105 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:52.414107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" event={"ID":"452f021b-1cd2-4a35-9b1b-89d6a0c533cb","Type":"ContainerStarted","Data":"8392688f8e4e1a6e55760593db20897feb31b74df701547102808e1eb39986fd"} Sep 15 21:13:52.432115 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:52.432063 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" podStartSLOduration=2.013003121 podStartE2EDuration="2.432046964s" totalImagesPullingTime="419.043843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:52.430872709 +0000 UTC m=+2221.560609294" watchObservedRunningTime="2026-09-15 21:13:52.432046964 +0000 UTC m=+2221.561783547" Sep 15 21:13:58.426232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:13:58.426199 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:14:12.271187 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:12.271145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:12.372575 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:12.372536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:13.109085 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:13.109048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:13.920482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:13.920437 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:14.571593 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:14.571554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:15.237321 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:15.237287 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:15.940578 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:15.940546 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:16.589473 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:16.589440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:17.335510 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:17.335472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:18.730186 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:18.730134 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:19.355229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.355192 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7"] Sep 15 21:14:19.355549 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.355484 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" podUID="452f021b-1cd2-4a35-9b1b-89d6a0c533cb" containerName="maas-api" containerID="cri-o://e474c9966c3286aec3916df3ba61a7200f1df45d3c912915c429d0621449eaf2" gracePeriod=30 Sep 15 21:14:19.381516 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.381479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:19.448596 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.448563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:19.552658 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.552621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:19.576804 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.576768 2580 generic.go:356] "Generic (PLEG): container finished" podID="452f021b-1cd2-4a35-9b1b-89d6a0c533cb" containerID="e474c9966c3286aec3916df3ba61a7200f1df45d3c912915c429d0621449eaf2" exitCode=0 Sep 15 21:14:19.576987 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.576824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" event={"ID":"452f021b-1cd2-4a35-9b1b-89d6a0c533cb","Type":"ContainerDied","Data":"e474c9966c3286aec3916df3ba61a7200f1df45d3c912915c429d0621449eaf2"} Sep 15 21:14:19.602435 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.602397 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:14:19.624615 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.624552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:19.680153 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.680128 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-metrics-certs\") pod \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " Sep 15 21:14:19.680317 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.680160 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls\") pod \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " Sep 15 21:14:19.680317 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.680188 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxh6c\" (UniqueName: \"kubernetes.io/projected/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-kube-api-access-sxh6c\") pod \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " Sep 15 21:14:19.680431 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.680364 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-openshift-service-ca\") pod \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\" (UID: \"452f021b-1cd2-4a35-9b1b-89d6a0c533cb\") " Sep 15 21:14:19.680784 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.680755 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "452f021b-1cd2-4a35-9b1b-89d6a0c533cb" (UID: "452f021b-1cd2-4a35-9b1b-89d6a0c533cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:14:19.682468 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.682444 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-kube-api-access-sxh6c" (OuterVolumeSpecName: "kube-api-access-sxh6c") pod "452f021b-1cd2-4a35-9b1b-89d6a0c533cb" (UID: "452f021b-1cd2-4a35-9b1b-89d6a0c533cb"). InnerVolumeSpecName "kube-api-access-sxh6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:14:19.682780 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.682758 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "452f021b-1cd2-4a35-9b1b-89d6a0c533cb" (UID: "452f021b-1cd2-4a35-9b1b-89d6a0c533cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:14:19.682838 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.682778 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "452f021b-1cd2-4a35-9b1b-89d6a0c533cb" (UID: "452f021b-1cd2-4a35-9b1b-89d6a0c533cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:14:19.781955 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.781918 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxh6c\" (UniqueName: \"kubernetes.io/projected/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-kube-api-access-sxh6c\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:19.781955 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.781951 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:19.782381 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.781964 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:19.782381 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.781976 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452f021b-1cd2-4a35-9b1b-89d6a0c533cb-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:19.942664 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:19.942578 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:20.051627 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:20.051574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:20.584774 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:20.584731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" event={"ID":"452f021b-1cd2-4a35-9b1b-89d6a0c533cb","Type":"ContainerDied","Data":"8392688f8e4e1a6e55760593db20897feb31b74df701547102808e1eb39986fd"} Sep 15 21:14:20.584950 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:20.584794 2580 scope.go:117] "RemoveContainer" containerID="e474c9966c3286aec3916df3ba61a7200f1df45d3c912915c429d0621449eaf2" Sep 15 21:14:20.584950 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:20.584752 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7" Sep 15 21:14:20.607091 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:20.607063 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7"] Sep 15 21:14:20.609649 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:20.609613 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a6f49f-78b6f6494b-4pqh7"] Sep 15 21:14:20.925341 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:20.925258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:21.314119 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:21.314087 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452f021b-1cd2-4a35-9b1b-89d6a0c533cb" path="/var/lib/kubelet/pods/452f021b-1cd2-4a35-9b1b-89d6a0c533cb/volumes" Sep 15 21:14:21.648318 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:21.648234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:22.329641 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:22.329605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:23.113855 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:23.113820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:25.766667 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:25.766633 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:25.834373 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:25.834340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:26.049682 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:26.049607 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:30.011307 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:30.011265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:30.841896 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:30.841865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:31.498931 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:31.498900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:32.178709 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:32.178665 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:32.943991 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:32.943957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:33.565621 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:33.565572 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:45.219215 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.219178 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842"] Sep 15 21:14:45.219742 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.219452 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" podUID="6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" containerName="maas-api" containerID="cri-o://c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d" gracePeriod=30 Sep 15 21:14:45.249500 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.249466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:45.319272 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.319232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:45.387887 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.387852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:45.457212 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.457168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:45.480981 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.480927 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:14:45.627201 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.627168 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-maas-api-tls\") pod \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " Sep 15 21:14:45.627390 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.627213 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-openshift-service-ca\") pod \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " Sep 15 21:14:45.627390 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.627241 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-metrics-certs\") pod \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " Sep 15 21:14:45.627390 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.627337 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d6dj\" (UniqueName: \"kubernetes.io/projected/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-kube-api-access-8d6dj\") pod \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\" (UID: \"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36\") " Sep 15 21:14:45.627747 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.627716 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" (UID: "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:14:45.629387 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.629358 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-kube-api-access-8d6dj" (OuterVolumeSpecName: "kube-api-access-8d6dj") pod "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" (UID: "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36"). InnerVolumeSpecName "kube-api-access-8d6dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:14:45.629835 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.629809 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" (UID: "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:14:45.629939 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.629830 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" (UID: "6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:14:45.728935 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.728903 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d6dj\" (UniqueName: \"kubernetes.io/projected/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-kube-api-access-8d6dj\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:45.729072 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.728939 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:45.729072 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.728955 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:45.729072 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.728969 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:14:45.735147 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.735068 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" containerID="c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d" exitCode=0 Sep 15 21:14:45.735147 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.735115 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" Sep 15 21:14:45.735303 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.735162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" event={"ID":"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36","Type":"ContainerDied","Data":"c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d"} Sep 15 21:14:45.735303 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.735214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842" event={"ID":"6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36","Type":"ContainerDied","Data":"3323fa55b82571e4a757405bf1ed1e5a597cf36f96f8c27a6a5d4058930107e8"} Sep 15 21:14:45.735303 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.735237 2580 scope.go:117] "RemoveContainer" containerID="c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d" Sep 15 21:14:45.744827 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.744809 2580 scope.go:117] "RemoveContainer" containerID="c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d" Sep 15 21:14:45.745082 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:14:45.745062 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d\": container with ID starting with c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d not found: ID does not exist" containerID="c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d" Sep 15 21:14:45.745126 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.745091 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d"} err="failed to get container status \"c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d\": rpc error: code = NotFound desc = could not find container \"c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d\": container with ID starting with c6b6d82591aabb19c8ed72719242ae1de953f5317267fffca36b73bd79a3fa8d not found: ID does not exist" Sep 15 21:14:45.755176 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.755148 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842"] Sep 15 21:14:45.762492 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.762472 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-219de8-d7f7fc544-r4842"] Sep 15 21:14:45.897958 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:45.897928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:46.019979 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:46.019924 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:46.633197 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:46.633168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:47.302341 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:47.302305 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:47.314298 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:47.314268 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" path="/var/lib/kubelet/pods/6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36/volumes" Sep 15 21:14:48.127533 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:48.127492 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:14:50.909685 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:14:50.909652 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:27.235012 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:27.234980 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:27.851535 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:27.851501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:33.584459 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:33.584401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:33.684346 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:33.684311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:33.806555 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:33.806512 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk"] Sep 15 21:15:33.806799 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:33.806772 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" containerID="cri-o://e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7" gracePeriod=30 Sep 15 21:15:33.830853 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:33.830819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:35.119019 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:35.118971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:35.918925 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:35.918889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:36.615648 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:36.615612 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:38.411214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:38.411174 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:39.059340 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.059310 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:15:39.075389 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.075318 2580 generic.go:356] "Generic (PLEG): container finished" podID="b331c713-9273-4db3-ae12-43159ab241b5" containerID="e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7" exitCode=0 Sep 15 21:15:39.075389 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.075363 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" Sep 15 21:15:39.075571 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.075365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" event={"ID":"b331c713-9273-4db3-ae12-43159ab241b5","Type":"ContainerDied","Data":"e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7"} Sep 15 21:15:39.075571 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.075459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk" event={"ID":"b331c713-9273-4db3-ae12-43159ab241b5","Type":"ContainerDied","Data":"dc547a23d30e0c41889354414221b89161ce3f046cb403b6dbf051b36cb48409"} Sep 15 21:15:39.075571 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.075487 2580 scope.go:117] "RemoveContainer" containerID="e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7" Sep 15 21:15:39.086199 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.086169 2580 scope.go:117] "RemoveContainer" containerID="2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a" Sep 15 21:15:39.097040 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.097022 2580 scope.go:117] "RemoveContainer" containerID="e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7" Sep 15 21:15:39.097305 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:15:39.097288 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7\": container with ID starting with e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7 not found: ID does not exist" containerID="e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7" Sep 15 21:15:39.097362 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.097317 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7"} err="failed to get container status \"e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7\": rpc error: code = NotFound desc = could not find container \"e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7\": container with ID starting with e443b6fe400b22908020ec28c2041d336b12b77d9b6cb60399372cb4d536dfc7 not found: ID does not exist" Sep 15 21:15:39.097362 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.097333 2580 scope.go:117] "RemoveContainer" containerID="2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a" Sep 15 21:15:39.097632 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:15:39.097618 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a\": container with ID starting with 2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a not found: ID does not exist" containerID="2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a" Sep 15 21:15:39.097686 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.097638 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a"} err="failed to get container status \"2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a\": rpc error: code = NotFound desc = could not find container \"2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a\": container with ID starting with 2a096b1fcd4cfaa859ebc5f4b4124343949619dff675515fb0d61d2970bb399a not found: ID does not exist" Sep 15 21:15:39.141162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141136 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-credential-socket\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141276 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b331c713-9273-4db3-ae12-43159ab241b5-istiod-ca-cert\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141276 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141234 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-socket\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141276 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141258 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-istio-token\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141465 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141293 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-certs\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141465 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141348 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-data\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141465 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141382 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgbrw\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-kube-api-access-cgbrw\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141465 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141382 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:39.141465 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141462 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-envoy\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141725 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141502 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b331c713-9273-4db3-ae12-43159ab241b5-istio-podinfo\") pod \"b331c713-9273-4db3-ae12-43159ab241b5\" (UID: \"b331c713-9273-4db3-ae12-43159ab241b5\") " Sep 15 21:15:39.141725 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141614 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:39.141725 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141672 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b331c713-9273-4db3-ae12-43159ab241b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:15:39.141725 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141685 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:39.141950 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141921 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:39.142017 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141951 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b331c713-9273-4db3-ae12-43159ab241b5-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.142017 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141971 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.142017 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141985 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.142017 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.141999 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.143490 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.143464 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:15:39.143689 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.143661 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:15:39.143764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.143666 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b331c713-9273-4db3-ae12-43159ab241b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:15:39.143764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.143722 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-kube-api-access-cgbrw" (OuterVolumeSpecName: "kube-api-access-cgbrw") pod "b331c713-9273-4db3-ae12-43159ab241b5" (UID: "b331c713-9273-4db3-ae12-43159ab241b5"). InnerVolumeSpecName "kube-api-access-cgbrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:15:39.243464 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.243439 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.243464 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.243462 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgbrw\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-kube-api-access-cgbrw\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.243595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.243473 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b331c713-9273-4db3-ae12-43159ab241b5-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.243595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.243481 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b331c713-9273-4db3-ae12-43159ab241b5-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.243595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.243489 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b331c713-9273-4db3-ae12-43159ab241b5-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:39.392513 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.392439 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk"] Sep 15 21:15:39.399386 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:39.399352 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-219de8-openshift-default-54d4cfd768-6ldzk"] Sep 15 21:15:41.313879 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:41.313844 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b331c713-9273-4db3-ae12-43159ab241b5" path="/var/lib/kubelet/pods/b331c713-9273-4db3-ae12-43159ab241b5/volumes" Sep 15 21:15:44.848512 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:44.848399 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng"] Sep 15 21:15:44.848961 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:44.848652 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" podUID="93b8c55d-cec6-4fc6-ad65-a61c82ed053a" containerName="maas-api" containerID="cri-o://f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661" gracePeriod=30 Sep 15 21:15:44.890731 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:44.890697 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:44.974606 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:44.974575 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:45.079305 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.079275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:45.098507 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.098475 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:15:45.110541 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.110477 2580 generic.go:356] "Generic (PLEG): container finished" podID="93b8c55d-cec6-4fc6-ad65-a61c82ed053a" containerID="f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661" exitCode=0 Sep 15 21:15:45.110675 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.110540 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" Sep 15 21:15:45.110675 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.110574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" event={"ID":"93b8c55d-cec6-4fc6-ad65-a61c82ed053a","Type":"ContainerDied","Data":"f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661"} Sep 15 21:15:45.110675 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.110622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng" event={"ID":"93b8c55d-cec6-4fc6-ad65-a61c82ed053a","Type":"ContainerDied","Data":"c025056bd261b14baed35d2f98220e88d6a7c42e8bf128f632197bcfebe2aa61"} Sep 15 21:15:45.110675 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.110664 2580 scope.go:117] "RemoveContainer" containerID="f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661" Sep 15 21:15:45.120382 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.120367 2580 scope.go:117] "RemoveContainer" containerID="f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661" Sep 15 21:15:45.120668 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:15:45.120651 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661\": container with ID starting with f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661 not found: ID does not exist" containerID="f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661" Sep 15 21:15:45.120725 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.120682 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661"} err="failed to get container status \"f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661\": rpc error: code = NotFound desc = could not find container \"f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661\": container with ID starting with f2349911114e30e80701a1265802a8e488274b08162117e3e4b300a4f4199661 not found: ID does not exist" Sep 15 21:15:45.172178 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.172152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:45.200325 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.200300 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-metrics-certs\") pod \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " Sep 15 21:15:45.200450 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.200353 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls\") pod \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " Sep 15 21:15:45.200450 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.200374 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7jkl\" (UniqueName: \"kubernetes.io/projected/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-kube-api-access-g7jkl\") pod \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " Sep 15 21:15:45.200556 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.200536 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-openshift-service-ca\") pod \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\" (UID: \"93b8c55d-cec6-4fc6-ad65-a61c82ed053a\") " Sep 15 21:15:45.200857 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.200832 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93b8c55d-cec6-4fc6-ad65-a61c82ed053a" (UID: "93b8c55d-cec6-4fc6-ad65-a61c82ed053a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:15:45.200961 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.200944 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:45.202499 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.202467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-kube-api-access-g7jkl" (OuterVolumeSpecName: "kube-api-access-g7jkl") pod "93b8c55d-cec6-4fc6-ad65-a61c82ed053a" (UID: "93b8c55d-cec6-4fc6-ad65-a61c82ed053a"). InnerVolumeSpecName "kube-api-access-g7jkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:15:45.202905 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.202882 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93b8c55d-cec6-4fc6-ad65-a61c82ed053a" (UID: "93b8c55d-cec6-4fc6-ad65-a61c82ed053a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:15:45.202905 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.202898 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93b8c55d-cec6-4fc6-ad65-a61c82ed053a" (UID: "93b8c55d-cec6-4fc6-ad65-a61c82ed053a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:15:45.301577 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.301541 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:45.301577 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.301570 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jkl\" (UniqueName: \"kubernetes.io/projected/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-kube-api-access-g7jkl\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:45.301577 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.301579 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b8c55d-cec6-4fc6-ad65-a61c82ed053a-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:15:45.429098 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.429023 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng"] Sep 15 21:15:45.431399 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.431378 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f09a7-66455b87d4-zqlng"] Sep 15 21:15:45.534509 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.534453 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:45.634675 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:45.634637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:46.326214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:46.326176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:47.007672 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:47.007635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:47.316576 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:47.316473 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b8c55d-cec6-4fc6-ad65-a61c82ed053a" path="/var/lib/kubelet/pods/93b8c55d-cec6-4fc6-ad65-a61c82ed053a/volumes" Sep 15 21:15:51.859585 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:51.859549 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:51.937123 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:51.937092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:52.134781 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:52.134749 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:52.759773 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:52.759736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:53.508227 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:53.508191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:54.156235 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:54.156198 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:15:54.850009 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:15:54.849971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:02.968628 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:02.968588 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78"] Sep 15 21:16:02.969169 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:02.968862 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" podUID="f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" containerName="maas-api" containerID="cri-o://7ae40698e5ddd6edc0836eaf8f672a0bd40332c25b394209d737b62fe66dddd3" gracePeriod=30 Sep 15 21:16:02.995229 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:02.995195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:03.065726 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.065674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:03.163243 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.163212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:03.219953 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.219885 2580 generic.go:356] "Generic (PLEG): container finished" podID="f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" containerID="7ae40698e5ddd6edc0836eaf8f672a0bd40332c25b394209d737b62fe66dddd3" exitCode=0 Sep 15 21:16:03.220073 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.219955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" event={"ID":"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4","Type":"ContainerDied","Data":"7ae40698e5ddd6edc0836eaf8f672a0bd40332c25b394209d737b62fe66dddd3"} Sep 15 21:16:03.220073 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.220002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" event={"ID":"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4","Type":"ContainerDied","Data":"ad012abc27494d40b44fc00597b9d7ae8f0145fb3c2a7d21a7dc51ae36404db1"} Sep 15 21:16:03.220073 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.220020 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad012abc27494d40b44fc00597b9d7ae8f0145fb3c2a7d21a7dc51ae36404db1" Sep 15 21:16:03.221991 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.221966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:03.226258 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.226241 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:16:03.373039 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.373003 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls\") pod \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " Sep 15 21:16:03.373287 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.373070 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-openshift-service-ca\") pod \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " Sep 15 21:16:03.373287 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.373133 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-metrics-certs\") pod \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " Sep 15 21:16:03.373287 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.373228 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bztkm\" (UniqueName: \"kubernetes.io/projected/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-kube-api-access-bztkm\") pod \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\" (UID: \"f86fb478-25eb-4e4d-81f0-b3bd71f77bb4\") " Sep 15 21:16:03.373501 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.373393 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" (UID: "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:16:03.373595 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.373572 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:03.375195 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.375167 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-kube-api-access-bztkm" (OuterVolumeSpecName: "kube-api-access-bztkm") pod "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" (UID: "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4"). InnerVolumeSpecName "kube-api-access-bztkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:03.375195 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.375183 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" (UID: "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:16:03.375316 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.375209 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" (UID: "f86fb478-25eb-4e4d-81f0-b3bd71f77bb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:16:03.474119 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.474043 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztkm\" (UniqueName: \"kubernetes.io/projected/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-kube-api-access-bztkm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:03.474119 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.474067 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:03.474119 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:03.474079 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:04.226191 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:04.226162 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78" Sep 15 21:16:04.250708 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:04.250652 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78"] Sep 15 21:16:04.255062 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:04.255036 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-280fb4-5d4ffcc9f6-x9f78"] Sep 15 21:16:04.738946 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:04.738904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:04.843773 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:04.843735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:05.315698 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:05.315666 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" path="/var/lib/kubelet/pods/f86fb478-25eb-4e4d-81f0-b3bd71f77bb4/volumes" Sep 15 21:16:09.562373 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:09.562339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:09.629781 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:09.629746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:09.831245 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:09.831159 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn"] Sep 15 21:16:09.831568 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:09.831539 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" containerID="cri-o://80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18" gracePeriod=30 Sep 15 21:16:09.836921 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:09.836860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:11.095221 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:11.095190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:11.729059 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:11.729019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:13.514363 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:13.514331 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:15.080857 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.080830 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:16:15.175700 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175624 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-certs\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.175700 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175671 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-envoy\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.175911 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175701 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-socket\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.175911 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175762 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-data\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.175911 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce890c77-7cf0-4bd6-9334-e45df65b141d-istiod-ca-cert\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.175911 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175832 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgmlh\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-kube-api-access-kgmlh\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.175911 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175896 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-credential-socket\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.176162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175901 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:15.176162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175930 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-podinfo\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.176162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.175966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-token\") pod \"ce890c77-7cf0-4bd6-9334-e45df65b141d\" (UID: \"ce890c77-7cf0-4bd6-9334-e45df65b141d\") " Sep 15 21:16:15.176162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176037 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:15.176162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176120 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:15.176162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176138 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:15.176482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176222 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce890c77-7cf0-4bd6-9334-e45df65b141d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:16:15.176482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176313 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.176482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176332 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.176482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176347 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.176482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176361 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce890c77-7cf0-4bd6-9334-e45df65b141d-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.176482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.176373 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.177971 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.177938 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:16:15.178180 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.178158 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-kube-api-access-kgmlh" (OuterVolumeSpecName: "kube-api-access-kgmlh") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "kube-api-access-kgmlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:15.178277 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.178178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:15.178277 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.178189 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce890c77-7cf0-4bd6-9334-e45df65b141d" (UID: "ce890c77-7cf0-4bd6-9334-e45df65b141d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 21:16:15.277016 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.276976 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.277016 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.277012 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.277016 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.277021 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce890c77-7cf0-4bd6-9334-e45df65b141d-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.277212 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.277030 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgmlh\" (UniqueName: \"kubernetes.io/projected/ce890c77-7cf0-4bd6-9334-e45df65b141d-kube-api-access-kgmlh\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:15.292947 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.292916 2580 generic.go:356] "Generic (PLEG): container finished" podID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerID="80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18" exitCode=0 Sep 15 21:16:15.293087 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.292955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" event={"ID":"ce890c77-7cf0-4bd6-9334-e45df65b141d","Type":"ContainerDied","Data":"80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18"} Sep 15 21:16:15.293087 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.292969 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" Sep 15 21:16:15.293087 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.293042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn" event={"ID":"ce890c77-7cf0-4bd6-9334-e45df65b141d","Type":"ContainerDied","Data":"3668a83b3a1215a493dd1402a179e21ea0d47154daf24e79eff1cb5977c60e47"} Sep 15 21:16:15.293087 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.293068 2580 scope.go:117] "RemoveContainer" containerID="80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18" Sep 15 21:16:15.302802 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.302782 2580 scope.go:117] "RemoveContainer" containerID="3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3" Sep 15 21:16:15.311749 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.311728 2580 scope.go:117] "RemoveContainer" containerID="80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18" Sep 15 21:16:15.312455 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:16:15.312427 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18\": container with ID starting with 80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18 not found: ID does not exist" containerID="80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18" Sep 15 21:16:15.312580 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.312465 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18"} err="failed to get container status \"80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18\": rpc error: code = NotFound desc = could not find container \"80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18\": container with ID starting with 80555c91327696f1433fe3b693a06856af6cdac285e42da5d6011c12212c0f18 not found: ID does not exist" Sep 15 21:16:15.312580 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.312482 2580 scope.go:117] "RemoveContainer" containerID="3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3" Sep 15 21:16:15.312764 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:16:15.312744 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3\": container with ID starting with 3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3 not found: ID does not exist" containerID="3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3" Sep 15 21:16:15.312809 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.312771 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3"} err="failed to get container status \"3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3\": rpc error: code = NotFound desc = could not find container \"3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3\": container with ID starting with 3850c5fcaa8c9116e96031195e0e4674404514f7e7bce9a58e5f55af4c122ae3 not found: ID does not exist" Sep 15 21:16:15.315101 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.315076 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn"] Sep 15 21:16:15.318439 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:15.318416 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-280fb4-openshift-default-5df96fc86d-x4pwn"] Sep 15 21:16:17.314192 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:17.314159 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" path="/var/lib/kubelet/pods/ce890c77-7cf0-4bd6-9334-e45df65b141d/volumes" Sep 15 21:16:20.541808 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:20.541746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:21.109988 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.109952 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm"] Sep 15 21:16:21.110277 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.110246 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" podUID="3f216cfa-94c5-487a-bac0-61d7dee333fa" containerName="maas-api" containerID="cri-o://bb6f51f5980070eb00607216c601807065c67a241b9d717d554a3c347957ae23" gracePeriod=30 Sep 15 21:16:21.191064 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.191026 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:21.275173 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.275127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:21.331736 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.331706 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f216cfa-94c5-487a-bac0-61d7dee333fa" containerID="bb6f51f5980070eb00607216c601807065c67a241b9d717d554a3c347957ae23" exitCode=0 Sep 15 21:16:21.331896 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.331776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" event={"ID":"3f216cfa-94c5-487a-bac0-61d7dee333fa","Type":"ContainerDied","Data":"bb6f51f5980070eb00607216c601807065c67a241b9d717d554a3c347957ae23"} Sep 15 21:16:21.377121 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.377096 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:16:21.377699 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.377679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:21.429486 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.429451 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhgsb\" (UniqueName: \"kubernetes.io/projected/3f216cfa-94c5-487a-bac0-61d7dee333fa-kube-api-access-vhgsb\") pod \"3f216cfa-94c5-487a-bac0-61d7dee333fa\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " Sep 15 21:16:21.429658 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.429498 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-metrics-certs\") pod \"3f216cfa-94c5-487a-bac0-61d7dee333fa\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " Sep 15 21:16:21.429658 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.429558 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f216cfa-94c5-487a-bac0-61d7dee333fa-openshift-service-ca\") pod \"3f216cfa-94c5-487a-bac0-61d7dee333fa\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " Sep 15 21:16:21.429658 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.429655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls\") pod \"3f216cfa-94c5-487a-bac0-61d7dee333fa\" (UID: \"3f216cfa-94c5-487a-bac0-61d7dee333fa\") " Sep 15 21:16:21.430166 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.429954 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f216cfa-94c5-487a-bac0-61d7dee333fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f216cfa-94c5-487a-bac0-61d7dee333fa" (UID: "3f216cfa-94c5-487a-bac0-61d7dee333fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:16:21.430166 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.430078 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f216cfa-94c5-487a-bac0-61d7dee333fa-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:21.431798 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.431768 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f216cfa-94c5-487a-bac0-61d7dee333fa-kube-api-access-vhgsb" (OuterVolumeSpecName: "kube-api-access-vhgsb") pod "3f216cfa-94c5-487a-bac0-61d7dee333fa" (UID: "3f216cfa-94c5-487a-bac0-61d7dee333fa"). InnerVolumeSpecName "kube-api-access-vhgsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:16:21.432158 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.432132 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f216cfa-94c5-487a-bac0-61d7dee333fa" (UID: "3f216cfa-94c5-487a-bac0-61d7dee333fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:16:21.432257 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.432153 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f216cfa-94c5-487a-bac0-61d7dee333fa" (UID: "3f216cfa-94c5-487a-bac0-61d7dee333fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:16:21.458302 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.458277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:21.530911 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.530872 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:21.530911 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.530904 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhgsb\" (UniqueName: \"kubernetes.io/projected/3f216cfa-94c5-487a-bac0-61d7dee333fa-kube-api-access-vhgsb\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:21.531091 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.530921 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f216cfa-94c5-487a-bac0-61d7dee333fa-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:16:21.828818 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.828779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:21.929047 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:21.929014 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:22.338915 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:22.338889 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" Sep 15 21:16:22.339094 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:22.338884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm" event={"ID":"3f216cfa-94c5-487a-bac0-61d7dee333fa","Type":"ContainerDied","Data":"82561cc8568612b2dceb89f0f633c04c632cf4731e07b75f881b6a9e5bf7057b"} Sep 15 21:16:22.339094 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:22.339020 2580 scope.go:117] "RemoveContainer" containerID="bb6f51f5980070eb00607216c601807065c67a241b9d717d554a3c347957ae23" Sep 15 21:16:22.364374 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:22.364349 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm"] Sep 15 21:16:22.369128 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:22.369106 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-728877-57dfcd558d-tw5nm"] Sep 15 21:16:23.315232 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:23.315199 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f216cfa-94c5-487a-bac0-61d7dee333fa" path="/var/lib/kubelet/pods/3f216cfa-94c5-487a-bac0-61d7dee333fa/volumes" Sep 15 21:16:28.126428 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:28.126369 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:28.228600 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:28.228559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:28.434917 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:28.434829 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:16:51.555217 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:51.555190 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:16:51.563058 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:51.563033 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:16:51.749717 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:16:51.749687 2580 scope.go:117] "RemoveContainer" containerID="7ae40698e5ddd6edc0836eaf8f672a0bd40332c25b394209d737b62fe66dddd3" Sep 15 21:17:19.467295 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:19.467264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:19.532019 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:19.531992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:29.561598 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:29.561545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.627080 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:32.627048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:32.727727 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:32.727660 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:33.755242 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755207 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-dk7rm"] Sep 15 21:17:33.755900 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755877 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755903 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755918 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755926 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755936 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755945 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755957 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f216cfa-94c5-487a-bac0-61d7dee333fa" containerName="maas-api" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755966 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f216cfa-94c5-487a-bac0-61d7dee333fa" containerName="maas-api" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755977 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452f021b-1cd2-4a35-9b1b-89d6a0c533cb" containerName="maas-api" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.755985 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="452f021b-1cd2-4a35-9b1b-89d6a0c533cb" containerName="maas-api" Sep 15 21:17:33.756000 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756003 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756011 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756019 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b8c55d-cec6-4fc6-ad65-a61c82ed053a" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756028 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b8c55d-cec6-4fc6-ad65-a61c82ed053a" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756043 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756051 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756076 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756084 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756191 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756205 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce890c77-7cf0-4bd6-9334-e45df65b141d" containerName="istio-proxy" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756215 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86fb478-25eb-4e4d-81f0-b3bd71f77bb4" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756224 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756231 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="452f021b-1cd2-4a35-9b1b-89d6a0c533cb" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756242 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f216cfa-94c5-487a-bac0-61d7dee333fa" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756250 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5b1ecb-cba5-4911-b6d2-2a2cf10e9f36" containerName="maas-api" Sep 15 21:17:33.756539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.756257 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b8c55d-cec6-4fc6-ad65-a61c82ed053a" containerName="maas-api" Sep 15 21:17:33.759708 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.759687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.762710 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.762686 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-shjx4\"" Sep 15 21:17:33.762710 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.762704 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 15 21:17:33.762855 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.762708 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 15 21:17:33.762855 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.762696 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 15 21:17:33.768276 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.767851 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-dk7rm"] Sep 15 21:17:33.842727 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.842698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-webhook-cert\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.842847 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.842735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-metrics-certs\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.842847 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.842825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkf7z\" (UniqueName: \"kubernetes.io/projected/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-kube-api-access-vkf7z\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.943680 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.943656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkf7z\" (UniqueName: \"kubernetes.io/projected/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-kube-api-access-vkf7z\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.943803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.943712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-webhook-cert\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.943803 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.943735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-metrics-certs\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.946089 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.946067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-metrics-certs\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.946191 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.946153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-webhook-cert\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:33.951121 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:33.951104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkf7z\" (UniqueName: \"kubernetes.io/projected/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-kube-api-access-vkf7z\") pod \"maas-controller-68d9b8fcfd-dk7rm\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:34.071488 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:34.071391 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:34.343797 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:34.343772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-dk7rm"] Sep 15 21:17:34.345263 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:17:34.345233 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a3c93f_0539_44f2_8f6f_b6981fae09eb.slice/crio-bb52905bf7a58323547906f22700c47b7581b0585f63f5cf2ab63aceda9e42c7 WatchSource:0}: Error finding container bb52905bf7a58323547906f22700c47b7581b0585f63f5cf2ab63aceda9e42c7: Status 404 returned error can't find the container with id bb52905bf7a58323547906f22700c47b7581b0585f63f5cf2ab63aceda9e42c7 Sep 15 21:17:34.346731 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:34.346712 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:17:34.754019 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:34.753982 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" event={"ID":"d7a3c93f-0539-44f2-8f6f-b6981fae09eb","Type":"ContainerStarted","Data":"bb52905bf7a58323547906f22700c47b7581b0585f63f5cf2ab63aceda9e42c7"} Sep 15 21:17:36.767598 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:36.767552 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" event={"ID":"d7a3c93f-0539-44f2-8f6f-b6981fae09eb","Type":"ContainerStarted","Data":"8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2"} Sep 15 21:17:36.767990 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:36.767694 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:36.783632 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:36.783579 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" podStartSLOduration=2.011839045 podStartE2EDuration="3.783559468s" totalImagesPullingTime="1.771720423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:36.782505761 +0000 UTC m=+2445.912242344" watchObservedRunningTime="2026-09-15 21:17:36.783559468 +0000 UTC m=+2445.913296078" Sep 15 21:17:47.775905 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:47.775877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:54.927586 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:54.927544 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:55.028451 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.028383 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-dk7rm"] Sep 15 21:17:55.028688 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.028641 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" podUID="d7a3c93f-0539-44f2-8f6f-b6981fae09eb" containerName="manager" containerID="cri-o://8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2" gracePeriod=30 Sep 15 21:17:55.283690 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.283664 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:55.430903 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.430873 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-webhook-cert\") pod \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " Sep 15 21:17:55.430903 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.430903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-metrics-certs\") pod \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " Sep 15 21:17:55.431153 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.430978 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkf7z\" (UniqueName: \"kubernetes.io/projected/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-kube-api-access-vkf7z\") pod \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\" (UID: \"d7a3c93f-0539-44f2-8f6f-b6981fae09eb\") " Sep 15 21:17:55.433003 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.432965 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d7a3c93f-0539-44f2-8f6f-b6981fae09eb" (UID: "d7a3c93f-0539-44f2-8f6f-b6981fae09eb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:17:55.433003 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.432982 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-kube-api-access-vkf7z" (OuterVolumeSpecName: "kube-api-access-vkf7z") pod "d7a3c93f-0539-44f2-8f6f-b6981fae09eb" (UID: "d7a3c93f-0539-44f2-8f6f-b6981fae09eb"). InnerVolumeSpecName "kube-api-access-vkf7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:17:55.433161 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.433062 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d7a3c93f-0539-44f2-8f6f-b6981fae09eb" (UID: "d7a3c93f-0539-44f2-8f6f-b6981fae09eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:17:55.531670 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.531639 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-webhook-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.531670 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.531665 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.531861 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.531675 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkf7z\" (UniqueName: \"kubernetes.io/projected/d7a3c93f-0539-44f2-8f6f-b6981fae09eb-kube-api-access-vkf7z\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:17:55.877573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.877466 2580 generic.go:356] "Generic (PLEG): container finished" podID="d7a3c93f-0539-44f2-8f6f-b6981fae09eb" containerID="8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2" exitCode=0 Sep 15 21:17:55.877573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.877530 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" Sep 15 21:17:55.877573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.877559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" event={"ID":"d7a3c93f-0539-44f2-8f6f-b6981fae09eb","Type":"ContainerDied","Data":"8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2"} Sep 15 21:17:55.877876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.877617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-dk7rm" event={"ID":"d7a3c93f-0539-44f2-8f6f-b6981fae09eb","Type":"ContainerDied","Data":"bb52905bf7a58323547906f22700c47b7581b0585f63f5cf2ab63aceda9e42c7"} Sep 15 21:17:55.877876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.877642 2580 scope.go:117] "RemoveContainer" containerID="8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2" Sep 15 21:17:55.888921 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.888719 2580 scope.go:117] "RemoveContainer" containerID="8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2" Sep 15 21:17:55.889177 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:17:55.889138 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2\": container with ID starting with 8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2 not found: ID does not exist" containerID="8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2" Sep 15 21:17:55.889286 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.889186 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2"} err="failed to get container status \"8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2\": rpc error: code = NotFound desc = could not find container \"8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2\": container with ID starting with 8d24390863a9caf0f457e9b5de21a1be42b62abb48521ab87c40397e0e3856a2 not found: ID does not exist" Sep 15 21:17:55.903698 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.903676 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-dk7rm"] Sep 15 21:17:55.908059 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:55.908038 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-dk7rm"] Sep 15 21:17:56.621021 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:56.620989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:17:57.314439 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:17:57.314387 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a3c93f-0539-44f2-8f6f-b6981fae09eb" path="/var/lib/kubelet/pods/d7a3c93f-0539-44f2-8f6f-b6981fae09eb/volumes" Sep 15 21:18:34.276461 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:18:34.276422 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:19:03.838001 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:03.837962 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns"] Sep 15 21:19:03.838559 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:03.838164 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerName="manager" containerID="cri-o://4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770" gracePeriod=10 Sep 15 21:19:04.183412 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.183372 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 21:19:04.220913 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.220886 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cf11874-73cc-4d79-9d9e-afbe42301acf-extensions-socket-volume\") pod \"2cf11874-73cc-4d79-9d9e-afbe42301acf\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " Sep 15 21:19:04.221081 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.220923 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtdqf\" (UniqueName: \"kubernetes.io/projected/2cf11874-73cc-4d79-9d9e-afbe42301acf-kube-api-access-mtdqf\") pod \"2cf11874-73cc-4d79-9d9e-afbe42301acf\" (UID: \"2cf11874-73cc-4d79-9d9e-afbe42301acf\") " Sep 15 21:19:04.221296 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.221271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf11874-73cc-4d79-9d9e-afbe42301acf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2cf11874-73cc-4d79-9d9e-afbe42301acf" (UID: "2cf11874-73cc-4d79-9d9e-afbe42301acf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 21:19:04.223138 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.223107 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf11874-73cc-4d79-9d9e-afbe42301acf-kube-api-access-mtdqf" (OuterVolumeSpecName: "kube-api-access-mtdqf") pod "2cf11874-73cc-4d79-9d9e-afbe42301acf" (UID: "2cf11874-73cc-4d79-9d9e-afbe42301acf"). InnerVolumeSpecName "kube-api-access-mtdqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:19:04.271206 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.271176 2580 generic.go:356] "Generic (PLEG): container finished" podID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerID="4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770" exitCode=0 Sep 15 21:19:04.271326 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.271219 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" Sep 15 21:19:04.271326 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.271253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" event={"ID":"2cf11874-73cc-4d79-9d9e-afbe42301acf","Type":"ContainerDied","Data":"4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770"} Sep 15 21:19:04.271326 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.271311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns" event={"ID":"2cf11874-73cc-4d79-9d9e-afbe42301acf","Type":"ContainerDied","Data":"7fcf2e2c17f635f73b91d90206b9c518e1f7adf04f1421de65683b677537b4a4"} Sep 15 21:19:04.271461 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.271344 2580 scope.go:117] "RemoveContainer" containerID="4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770" Sep 15 21:19:04.281511 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.281491 2580 scope.go:117] "RemoveContainer" containerID="031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3" Sep 15 21:19:04.289841 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.289824 2580 scope.go:117] "RemoveContainer" containerID="4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770" Sep 15 21:19:04.290084 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:19:04.290065 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770\": container with ID starting with 4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770 not found: ID does not exist" containerID="4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770" Sep 15 21:19:04.290123 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.290093 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770"} err="failed to get container status \"4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770\": rpc error: code = NotFound desc = could not find container \"4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770\": container with ID starting with 4d69ae87057f4d7fc899e55ad55cee0a96008153def7c3484c7aba3b6f3bc770 not found: ID does not exist" Sep 15 21:19:04.290123 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.290114 2580 scope.go:117] "RemoveContainer" containerID="031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3" Sep 15 21:19:04.290329 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:19:04.290314 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3\": container with ID starting with 031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3 not found: ID does not exist" containerID="031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3" Sep 15 21:19:04.290376 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.290332 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3"} err="failed to get container status \"031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3\": rpc error: code = NotFound desc = could not find container \"031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3\": container with ID starting with 031d0a9763dc5e34afaac5e64d569cf02fa3c57c11333b70da607dac39d85af3 not found: ID does not exist" Sep 15 21:19:04.294457 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.294435 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns"] Sep 15 21:19:04.298350 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.298331 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pxrns"] Sep 15 21:19:04.322251 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.322228 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cf11874-73cc-4d79-9d9e-afbe42301acf-extensions-socket-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:19:04.322251 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:04.322250 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtdqf\" (UniqueName: \"kubernetes.io/projected/2cf11874-73cc-4d79-9d9e-afbe42301acf-kube-api-access-mtdqf\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:19:05.314031 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:19:05.313995 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" path="/var/lib/kubelet/pods/2cf11874-73cc-4d79-9d9e-afbe42301acf/volumes" Sep 15 21:20:04.554870 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.554831 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc"] Sep 15 21:20:04.555303 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555256 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerName="manager" Sep 15 21:20:04.555303 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555269 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerName="manager" Sep 15 21:20:04.555303 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555282 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a3c93f-0539-44f2-8f6f-b6981fae09eb" containerName="manager" Sep 15 21:20:04.555303 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555291 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a3c93f-0539-44f2-8f6f-b6981fae09eb" containerName="manager" Sep 15 21:20:04.555478 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555367 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a3c93f-0539-44f2-8f6f-b6981fae09eb" containerName="manager" Sep 15 21:20:04.555478 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555382 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerName="manager" Sep 15 21:20:04.555478 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555388 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b331c713-9273-4db3-ae12-43159ab241b5" containerName="istio-proxy" Sep 15 21:20:04.555478 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.555394 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerName="manager" Sep 15 21:20:04.558469 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.558454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:04.560849 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.560827 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rp6ch\"" Sep 15 21:20:04.569900 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.569877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc"] Sep 15 21:20:04.632093 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.632061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01b890a1-f6b1-49e8-8992-8448e084e534-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-62hvc\" (UID: \"01b890a1-f6b1-49e8-8992-8448e084e534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:04.632093 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.632097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn5lp\" (UniqueName: \"kubernetes.io/projected/01b890a1-f6b1-49e8-8992-8448e084e534-kube-api-access-cn5lp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-62hvc\" (UID: \"01b890a1-f6b1-49e8-8992-8448e084e534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:04.733122 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.733092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01b890a1-f6b1-49e8-8992-8448e084e534-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-62hvc\" (UID: \"01b890a1-f6b1-49e8-8992-8448e084e534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:04.733122 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.733123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn5lp\" (UniqueName: \"kubernetes.io/projected/01b890a1-f6b1-49e8-8992-8448e084e534-kube-api-access-cn5lp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-62hvc\" (UID: \"01b890a1-f6b1-49e8-8992-8448e084e534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:04.733541 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.733519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01b890a1-f6b1-49e8-8992-8448e084e534-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-62hvc\" (UID: \"01b890a1-f6b1-49e8-8992-8448e084e534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:04.748048 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.748023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn5lp\" (UniqueName: \"kubernetes.io/projected/01b890a1-f6b1-49e8-8992-8448e084e534-kube-api-access-cn5lp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-62hvc\" (UID: \"01b890a1-f6b1-49e8-8992-8448e084e534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:04.869541 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:04.869471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:05.158513 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:05.158484 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc"] Sep 15 21:20:05.160082 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:20:05.160058 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b890a1_f6b1_49e8_8992_8448e084e534.slice/crio-1fe6eb807ebd2151b20f04762fef54f3547e1a823ba12e57a1b30af5b9bde93a WatchSource:0}: Error finding container 1fe6eb807ebd2151b20f04762fef54f3547e1a823ba12e57a1b30af5b9bde93a: Status 404 returned error can't find the container with id 1fe6eb807ebd2151b20f04762fef54f3547e1a823ba12e57a1b30af5b9bde93a Sep 15 21:20:05.609486 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:05.609442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" event={"ID":"01b890a1-f6b1-49e8-8992-8448e084e534","Type":"ContainerStarted","Data":"d5e92d2b3d33cbb9c491898d5f0927e7ea87d84c694bde3304ac6ef9e04626f6"} Sep 15 21:20:05.609895 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:05.609592 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:05.609895 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:05.609624 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" event={"ID":"01b890a1-f6b1-49e8-8992-8448e084e534","Type":"ContainerStarted","Data":"1fe6eb807ebd2151b20f04762fef54f3547e1a823ba12e57a1b30af5b9bde93a"} Sep 15 21:20:05.636949 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:05.636899 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" podStartSLOduration=1.636885456 podStartE2EDuration="1.636885456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:05.632765087 +0000 UTC m=+2594.762501666" watchObservedRunningTime="2026-09-15 21:20:05.636885456 +0000 UTC m=+2594.766622036" Sep 15 21:20:16.618199 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:16.618168 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-62hvc" Sep 15 21:20:43.343382 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:20:43.343348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:06.301398 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:06.301354 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:07.721065 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:07.721028 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:29.742394 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:29.742363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:30.677600 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:30.677567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:21:48.317500 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.317461 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-tsxdb"] Sep 15 21:21:48.318140 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.317870 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerName="manager" Sep 15 21:21:48.318140 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.317882 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf11874-73cc-4d79-9d9e-afbe42301acf" containerName="manager" Sep 15 21:21:48.321189 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.321166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.324241 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.324216 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-crdcx\"" Sep 15 21:21:48.324700 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.324680 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 15 21:21:48.324900 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.324863 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 21:21:48.325009 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.324783 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 21:21:48.325074 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.324764 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 21:21:48.335634 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.335606 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-569cf8d545-dllk7"] Sep 15 21:21:48.339712 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.339687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-tsxdb"] Sep 15 21:21:48.339831 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.339808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.352820 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.352793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-569cf8d545-dllk7"] Sep 15 21:21:48.426521 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjs2\" (UniqueName: \"kubernetes.io/projected/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-kube-api-access-xfjs2\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.426690 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfq4z\" (UniqueName: \"kubernetes.io/projected/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-kube-api-access-rfq4z\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.426690 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-metrics-certs\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.426776 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-metrics-certs\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.426776 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-maas-api-tls\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.426872 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-maas-api-tls\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.426872 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-openshift-service-ca\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.426872 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.426854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-openshift-service-ca\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.528204 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjs2\" (UniqueName: \"kubernetes.io/projected/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-kube-api-access-xfjs2\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.528398 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfq4z\" (UniqueName: \"kubernetes.io/projected/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-kube-api-access-rfq4z\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.528398 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-metrics-certs\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.528398 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-metrics-certs\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.528398 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-maas-api-tls\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.528398 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-maas-api-tls\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.528721 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-openshift-service-ca\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.528721 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.528482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-openshift-service-ca\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.529764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.529323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-openshift-service-ca\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.529764 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.529700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-openshift-service-ca\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.531871 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.531841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-metrics-certs\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.532072 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.532049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-maas-api-tls\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.532702 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.532676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-metrics-certs\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.533345 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.533318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-maas-api-tls\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.536060 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.536034 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjs2\" (UniqueName: \"kubernetes.io/projected/8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7-kube-api-access-xfjs2\") pod \"maas-api-569cf8d545-dllk7\" (UID: \"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.536542 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.536523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfq4z\" (UniqueName: \"kubernetes.io/projected/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-kube-api-access-rfq4z\") pod \"maas-api-746c646f77-tsxdb\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.633030 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.632965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:48.656208 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.656182 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:48.915079 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.915050 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-tsxdb"] Sep 15 21:21:48.917227 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:21:48.917199 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfa7f59_2a35_4328_b8d5_1ca5477bb507.slice/crio-8f12b3fdf96441df9c5cdab1838e96665f0b47042a00268529ad8fa2fc44a597 WatchSource:0}: Error finding container 8f12b3fdf96441df9c5cdab1838e96665f0b47042a00268529ad8fa2fc44a597: Status 404 returned error can't find the container with id 8f12b3fdf96441df9c5cdab1838e96665f0b47042a00268529ad8fa2fc44a597 Sep 15 21:21:48.952194 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:48.952173 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-569cf8d545-dllk7"] Sep 15 21:21:48.953443 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:21:48.953396 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0402a0_b4ca_4d3e_8752_fe63fb9bfff7.slice/crio-071585d6fbff8055dc2522bd107d16715b327228e1414653b01c67d1acb69935 WatchSource:0}: Error finding container 071585d6fbff8055dc2522bd107d16715b327228e1414653b01c67d1acb69935: Status 404 returned error can't find the container with id 071585d6fbff8055dc2522bd107d16715b327228e1414653b01c67d1acb69935 Sep 15 21:21:49.214851 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:49.214755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" event={"ID":"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7","Type":"ContainerStarted","Data":"071585d6fbff8055dc2522bd107d16715b327228e1414653b01c67d1acb69935"} Sep 15 21:21:49.216178 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:49.216150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" event={"ID":"9bfa7f59-2a35-4328-b8d5-1ca5477bb507","Type":"ContainerStarted","Data":"8f12b3fdf96441df9c5cdab1838e96665f0b47042a00268529ad8fa2fc44a597"} Sep 15 21:21:50.225051 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:50.225004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" event={"ID":"8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7","Type":"ContainerStarted","Data":"d6484756252a9ecda95a1256e26306a3d4d7c4c6090e275166a37edf92516f1f"} Sep 15 21:21:50.225507 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:50.225112 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:50.226762 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:50.226731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" event={"ID":"9bfa7f59-2a35-4328-b8d5-1ca5477bb507","Type":"ContainerStarted","Data":"f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740"} Sep 15 21:21:50.226876 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:50.226823 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:50.241763 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:50.241692 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" podStartSLOduration=1.595208926 podStartE2EDuration="2.241680028s" totalImagesPullingTime="646.471102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:50.239614561 +0000 UTC m=+2699.369351144" watchObservedRunningTime="2026-09-15 21:21:50.241680028 +0000 UTC m=+2699.371416611" Sep 15 21:21:50.257539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:50.257502 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" podStartSLOduration=1.727383656 podStartE2EDuration="2.257489333s" totalImagesPullingTime="530.105677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:50.256134447 +0000 UTC m=+2699.385871022" watchObservedRunningTime="2026-09-15 21:21:50.257489333 +0000 UTC m=+2699.387225915" Sep 15 21:21:51.596560 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:51.596529 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:21:51.606039 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:51.606010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:21:56.239363 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.239321 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:56.239949 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.239729 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-569cf8d545-dllk7" Sep 15 21:21:56.303341 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.303304 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-tsxdb"] Sep 15 21:21:56.303806 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.303769 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" podUID="9bfa7f59-2a35-4328-b8d5-1ca5477bb507" containerName="maas-api" containerID="cri-o://f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740" gracePeriod=30 Sep 15 21:21:56.561173 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.561144 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:56.605919 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.605889 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-metrics-certs\") pod \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " Sep 15 21:21:56.606058 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.606009 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-maas-api-tls\") pod \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " Sep 15 21:21:56.606058 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.606038 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-openshift-service-ca\") pod \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " Sep 15 21:21:56.606166 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.606084 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfq4z\" (UniqueName: \"kubernetes.io/projected/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-kube-api-access-rfq4z\") pod \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\" (UID: \"9bfa7f59-2a35-4328-b8d5-1ca5477bb507\") " Sep 15 21:21:56.606447 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.606397 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9bfa7f59-2a35-4328-b8d5-1ca5477bb507" (UID: "9bfa7f59-2a35-4328-b8d5-1ca5477bb507"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 21:21:56.608261 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.608233 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9bfa7f59-2a35-4328-b8d5-1ca5477bb507" (UID: "9bfa7f59-2a35-4328-b8d5-1ca5477bb507"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:21:56.608500 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.608468 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9bfa7f59-2a35-4328-b8d5-1ca5477bb507" (UID: "9bfa7f59-2a35-4328-b8d5-1ca5477bb507"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 21:21:56.608590 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.608500 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-kube-api-access-rfq4z" (OuterVolumeSpecName: "kube-api-access-rfq4z") pod "9bfa7f59-2a35-4328-b8d5-1ca5477bb507" (UID: "9bfa7f59-2a35-4328-b8d5-1ca5477bb507"). InnerVolumeSpecName "kube-api-access-rfq4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 21:21:56.707156 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.707130 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:21:56.707156 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.707154 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:21:56.707302 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.707164 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfq4z\" (UniqueName: \"kubernetes.io/projected/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-kube-api-access-rfq4z\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:21:56.707302 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:56.707173 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfa7f59-2a35-4328-b8d5-1ca5477bb507-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 15 21:21:57.271328 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.271294 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bfa7f59-2a35-4328-b8d5-1ca5477bb507" containerID="f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740" exitCode=0 Sep 15 21:21:57.271771 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.271338 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" Sep 15 21:21:57.271771 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.271342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" event={"ID":"9bfa7f59-2a35-4328-b8d5-1ca5477bb507","Type":"ContainerDied","Data":"f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740"} Sep 15 21:21:57.271771 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.271468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-tsxdb" event={"ID":"9bfa7f59-2a35-4328-b8d5-1ca5477bb507","Type":"ContainerDied","Data":"8f12b3fdf96441df9c5cdab1838e96665f0b47042a00268529ad8fa2fc44a597"} Sep 15 21:21:57.271771 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.271491 2580 scope.go:117] "RemoveContainer" containerID="f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740" Sep 15 21:21:57.281455 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.281436 2580 scope.go:117] "RemoveContainer" containerID="f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740" Sep 15 21:21:57.281728 ip-10-0-139-201 kubenswrapper[2580]: E0915 21:21:57.281711 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740\": container with ID starting with f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740 not found: ID does not exist" containerID="f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740" Sep 15 21:21:57.281784 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.281735 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740"} err="failed to get container status \"f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740\": rpc error: code = NotFound desc = could not find container \"f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740\": container with ID starting with f7d175ae513c9e9dfaafedb651b1e31f8a49b32cc7b83b4bb63a74a1acbe7740 not found: ID does not exist" Sep 15 21:21:57.292378 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.292355 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-tsxdb"] Sep 15 21:21:57.297877 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.297857 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-tsxdb"] Sep 15 21:21:57.314743 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:21:57.314719 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bfa7f59-2a35-4328-b8d5-1ca5477bb507" path="/var/lib/kubelet/pods/9bfa7f59-2a35-4328-b8d5-1ca5477bb507/volumes" Sep 15 21:22:03.224748 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:03.224705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:20.876863 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:20.876808 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:27.579849 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:27.579816 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8bg5g_f04b6918-b1ee-41ed-989f-a187aee1e26a/manager/0.log" Sep 15 21:22:27.831166 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:27.831092 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2p2gd_b8ef9dcd-5b1c-487b-a779-b551c321365c/server/0.log" Sep 15 21:22:27.958144 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:27.958118 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-w7hvt_d3a611af-d53e-409c-a463-63b98690a3b2/manager/0.log" Sep 15 21:22:28.086619 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:28.086533 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-4fgfr_69d81a70-c5a5-4b80-9b20-cee452b4e24c/copy-manifests/0.log" Sep 15 21:22:28.098687 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:28.098647 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-4fgfr_69d81a70-c5a5-4b80-9b20-cee452b4e24c/manager/0.log" Sep 15 21:22:28.722245 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:28.722211 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-569cf8d545-dllk7_8b0402a0-b4ca-4d3e-8752-fe63fb9bfff7/maas-api/0.log" Sep 15 21:22:30.414330 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:30.414299 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-p74c5_3fc39bbf-541f-4898-921c-333b939a1106/kuadrant-console-plugin/0.log" Sep 15 21:22:30.542609 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:30.542576 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-62hvc_01b890a1-f6b1-49e8-8992-8448e084e534/manager/0.log" Sep 15 21:22:31.183901 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:31.183868 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l_1364f2e9-1cb6-4810-9daf-99fd905f06ab/istio-proxy/0.log" Sep 15 21:22:31.664519 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:31.664488 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kwffw_2e53a04c-0eeb-4627-962b-a4fbb8dff2fe/istio-proxy/1.log" Sep 15 21:22:31.785626 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:31.785584 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lqfgw_47e44d4f-614f-4962-8021-1b5dbc4edc74/payload-pre-processing/0.log" Sep 15 21:22:32.038285 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:32.038259 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f448b5b8-4bm9g_4bbf573e-e3c2-4d9d-8711-5a6cf974c302/router/0.log" Sep 15 21:22:32.632858 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:32.632811 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-5t8c4_06ba2a11-64d6-4e25-b35c-82b3b7a1f000/main/0.log" Sep 15 21:22:38.721794 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.721762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcx6s/must-gather-qjzqw"] Sep 15 21:22:38.722207 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.722169 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bfa7f59-2a35-4328-b8d5-1ca5477bb507" containerName="maas-api" Sep 15 21:22:38.722207 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.722180 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bfa7f59-2a35-4328-b8d5-1ca5477bb507" containerName="maas-api" Sep 15 21:22:38.722282 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.722243 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bfa7f59-2a35-4328-b8d5-1ca5477bb507" containerName="maas-api" Sep 15 21:22:38.725439 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.725417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:38.727604 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.727573 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tcx6s\"/\"default-dockercfg-8qfvh\"" Sep 15 21:22:38.728131 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.728109 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcx6s\"/\"openshift-service-ca.crt\"" Sep 15 21:22:38.728235 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.728112 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcx6s\"/\"kube-root-ca.crt\"" Sep 15 21:22:38.734663 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.734636 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/must-gather-qjzqw"] Sep 15 21:22:38.736174 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.736154 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:38.787070 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.787042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glljq\" (UniqueName: \"kubernetes.io/projected/3afec931-2f48-4942-aa75-0928d6c6d923-kube-api-access-glljq\") pod \"must-gather-qjzqw\" (UID: \"3afec931-2f48-4942-aa75-0928d6c6d923\") " pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:38.787183 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.787077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3afec931-2f48-4942-aa75-0928d6c6d923-must-gather-output\") pod \"must-gather-qjzqw\" (UID: \"3afec931-2f48-4942-aa75-0928d6c6d923\") " pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:38.887762 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.887734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glljq\" (UniqueName: \"kubernetes.io/projected/3afec931-2f48-4942-aa75-0928d6c6d923-kube-api-access-glljq\") pod \"must-gather-qjzqw\" (UID: \"3afec931-2f48-4942-aa75-0928d6c6d923\") " pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:38.887873 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.887767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3afec931-2f48-4942-aa75-0928d6c6d923-must-gather-output\") pod \"must-gather-qjzqw\" (UID: \"3afec931-2f48-4942-aa75-0928d6c6d923\") " pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:38.888038 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.888022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3afec931-2f48-4942-aa75-0928d6c6d923-must-gather-output\") pod \"must-gather-qjzqw\" (UID: \"3afec931-2f48-4942-aa75-0928d6c6d923\") " pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:38.895170 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:38.895140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glljq\" (UniqueName: \"kubernetes.io/projected/3afec931-2f48-4942-aa75-0928d6c6d923-kube-api-access-glljq\") pod \"must-gather-qjzqw\" (UID: \"3afec931-2f48-4942-aa75-0928d6c6d923\") " pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:39.029254 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:39.029227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:39.035140 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:39.035123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/must-gather-qjzqw" Sep 15 21:22:39.305074 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:39.305047 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/must-gather-qjzqw"] Sep 15 21:22:39.306489 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:22:39.306459 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3afec931_2f48_4942_aa75_0928d6c6d923.slice/crio-9710ee379866ed9b9b9b16ad0273a2f7af64b890d564880c181bdb303a32cd5d WatchSource:0}: Error finding container 9710ee379866ed9b9b9b16ad0273a2f7af64b890d564880c181bdb303a32cd5d: Status 404 returned error can't find the container with id 9710ee379866ed9b9b9b16ad0273a2f7af64b890d564880c181bdb303a32cd5d Sep 15 21:22:39.308562 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:39.308538 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 21:22:39.508373 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:39.508340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/must-gather-qjzqw" event={"ID":"3afec931-2f48-4942-aa75-0928d6c6d923","Type":"ContainerStarted","Data":"9710ee379866ed9b9b9b16ad0273a2f7af64b890d564880c181bdb303a32cd5d"} Sep 15 21:22:40.515717 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:40.515679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/must-gather-qjzqw" event={"ID":"3afec931-2f48-4942-aa75-0928d6c6d923","Type":"ContainerStarted","Data":"4d7185904ed50038641989a5b2d737c56fee4e90ad36b087a234a84a48de51c2"} Sep 15 21:22:40.516108 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:40.515733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/must-gather-qjzqw" event={"ID":"3afec931-2f48-4942-aa75-0928d6c6d923","Type":"ContainerStarted","Data":"a77d1b0e8ff9ddc342b0e33be8be7dc373670a3168a7de3343bebdabf5ad5010"} Sep 15 21:22:40.530897 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:40.530843 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcx6s/must-gather-qjzqw" podStartSLOduration=1.829565633 podStartE2EDuration="2.530826536s" totalImagesPullingTime="701.260903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:40.528926349 +0000 UTC m=+2749.658662960" watchObservedRunningTime="2026-09-15 21:22:40.530826536 +0000 UTC m=+2749.660563119" Sep 15 21:22:40.970969 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:40.970887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:41.071123 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:41.071090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" limit=1024 Sep 15 21:22:41.824475 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:41.824445 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6v4bn_73b7914e-e824-44f9-8a97-d735b8de33a7/global-pull-secret-syncer/0.log" Sep 15 21:22:41.947539 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:41.947509 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7tjzq_97adad06-f0b5-4467-b20f-9dafdc3058a6/konnectivity-agent/0.log" Sep 15 21:22:42.084442 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:42.084358 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-201.ec2.internal_b23ec63d9b3d220644ea926d481ee00f/haproxy/0.log" Sep 15 21:22:46.915680 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:46.915643 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-p74c5_3fc39bbf-541f-4898-921c-333b939a1106/kuadrant-console-plugin/0.log" Sep 15 21:22:46.953798 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:46.953767 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-62hvc_01b890a1-f6b1-49e8-8992-8448e084e534/manager/0.log" Sep 15 21:22:48.726205 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.726059 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03425d96-de2e-4bb4-9c24-67e304014b62/alertmanager/0.log" Sep 15 21:22:48.747549 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.747509 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03425d96-de2e-4bb4-9c24-67e304014b62/config-reloader/0.log" Sep 15 21:22:48.776202 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.776168 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03425d96-de2e-4bb4-9c24-67e304014b62/kube-rbac-proxy-web/0.log" Sep 15 21:22:48.795976 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.795909 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03425d96-de2e-4bb4-9c24-67e304014b62/kube-rbac-proxy/0.log" Sep 15 21:22:48.817957 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.817928 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03425d96-de2e-4bb4-9c24-67e304014b62/kube-rbac-proxy-metric/0.log" Sep 15 21:22:48.844573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.844540 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03425d96-de2e-4bb4-9c24-67e304014b62/prom-label-proxy/0.log" Sep 15 21:22:48.863990 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.863949 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03425d96-de2e-4bb4-9c24-67e304014b62/init-config-reloader/0.log" Sep 15 21:22:48.953995 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.953940 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rzhwn_3b963fbb-4fc0-4324-afeb-a43b23a5fb06/kube-state-metrics/0.log" Sep 15 21:22:48.975138 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.975111 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rzhwn_3b963fbb-4fc0-4324-afeb-a43b23a5fb06/kube-rbac-proxy-main/0.log" Sep 15 21:22:48.997665 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:48.997589 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rzhwn_3b963fbb-4fc0-4324-afeb-a43b23a5fb06/kube-rbac-proxy-self/0.log" Sep 15 21:22:49.155665 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.155625 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fq67k_0b05402b-6f75-451a-8bfe-296666a116c9/node-exporter/0.log" Sep 15 21:22:49.175385 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.175347 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fq67k_0b05402b-6f75-451a-8bfe-296666a116c9/kube-rbac-proxy/0.log" Sep 15 21:22:49.195312 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.195285 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fq67k_0b05402b-6f75-451a-8bfe-296666a116c9/init-textfile/0.log" Sep 15 21:22:49.318857 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.318820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-swb7j_5453c7e3-cd47-426b-8f3c-b8b88b555b79/kube-rbac-proxy-main/0.log" Sep 15 21:22:49.346207 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.346173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-swb7j_5453c7e3-cd47-426b-8f3c-b8b88b555b79/kube-rbac-proxy-self/0.log" Sep 15 21:22:49.387326 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.387297 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-swb7j_5453c7e3-cd47-426b-8f3c-b8b88b555b79/openshift-state-metrics/0.log" Sep 15 21:22:49.667573 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.667489 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66bb65b95-rdm5m_c6e0c9bd-9ee5-4226-ae50-3a571fb52d69/telemeter-client/0.log" Sep 15 21:22:49.691391 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.691360 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66bb65b95-rdm5m_c6e0c9bd-9ee5-4226-ae50-3a571fb52d69/reload/0.log" Sep 15 21:22:49.711918 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.711887 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66bb65b95-rdm5m_c6e0c9bd-9ee5-4226-ae50-3a571fb52d69/kube-rbac-proxy/0.log" Sep 15 21:22:49.743281 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.743251 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898449498-9xrh5_c1bf3427-37b9-4d9f-ad60-bbf6fbf21072/thanos-query/0.log" Sep 15 21:22:49.763747 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.763710 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898449498-9xrh5_c1bf3427-37b9-4d9f-ad60-bbf6fbf21072/kube-rbac-proxy-web/0.log" Sep 15 21:22:49.789546 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.789516 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898449498-9xrh5_c1bf3427-37b9-4d9f-ad60-bbf6fbf21072/kube-rbac-proxy/0.log" Sep 15 21:22:49.811581 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.811549 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898449498-9xrh5_c1bf3427-37b9-4d9f-ad60-bbf6fbf21072/prom-label-proxy/0.log" Sep 15 21:22:49.829594 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.829562 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898449498-9xrh5_c1bf3427-37b9-4d9f-ad60-bbf6fbf21072/kube-rbac-proxy-rules/0.log" Sep 15 21:22:49.853165 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:49.853135 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898449498-9xrh5_c1bf3427-37b9-4d9f-ad60-bbf6fbf21072/kube-rbac-proxy-metrics/0.log" Sep 15 21:22:51.741548 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.741506 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/2.log" Sep 15 21:22:51.748683 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.748655 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zzs5_057261c3-367a-40b0-ba97-328f74076ee0/console-operator/3.log" Sep 15 21:22:51.762916 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.762882 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl"] Sep 15 21:22:51.771723 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.771689 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:51.774993 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.774950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl"] Sep 15 21:22:51.940426 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.940371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-sys\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:51.940611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.940437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-podres\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:51.940611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.940514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8fj\" (UniqueName: \"kubernetes.io/projected/f0bc13d0-4d2b-415e-8d73-31d759124b81-kube-api-access-tg8fj\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:51.940611 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.940539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-lib-modules\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:51.940794 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:51.940636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-proc\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042080 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-sys\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042280 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-podres\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042280 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8fj\" (UniqueName: \"kubernetes.io/projected/f0bc13d0-4d2b-415e-8d73-31d759124b81-kube-api-access-tg8fj\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042280 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-lib-modules\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042480 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-sys\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042619 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042596 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-lib-modules\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042918 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-proc\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.042999 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-podres\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.043058 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.042777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0bc13d0-4d2b-415e-8d73-31d759124b81-proc\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.051422 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.051342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8fj\" (UniqueName: \"kubernetes.io/projected/f0bc13d0-4d2b-415e-8d73-31d759124b81-kube-api-access-tg8fj\") pod \"perf-node-gather-daemonset-nkstl\" (UID: \"f0bc13d0-4d2b-415e-8d73-31d759124b81\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.088214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.088181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.381464 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.381427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl"] Sep 15 21:22:52.384462 ip-10-0-139-201 kubenswrapper[2580]: W0915 21:22:52.384429 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf0bc13d0_4d2b_415e_8d73_31d759124b81.slice/crio-b880bacd5ea602ea9b4ed5171f98c3b50d7dca594f030688249ed4698db93e85 WatchSource:0}: Error finding container b880bacd5ea602ea9b4ed5171f98c3b50d7dca594f030688249ed4698db93e85: Status 404 returned error can't find the container with id b880bacd5ea602ea9b4ed5171f98c3b50d7dca594f030688249ed4698db93e85 Sep 15 21:22:52.663482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.662491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" event={"ID":"f0bc13d0-4d2b-415e-8d73-31d759124b81","Type":"ContainerStarted","Data":"04e8a1fe50c872867594229083a4647e06041799c8c6d06b9327c68e8ddd10a5"} Sep 15 21:22:52.663482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.662577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" event={"ID":"f0bc13d0-4d2b-415e-8d73-31d759124b81","Type":"ContainerStarted","Data":"b880bacd5ea602ea9b4ed5171f98c3b50d7dca594f030688249ed4698db93e85"} Sep 15 21:22:52.663482 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.663363 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:22:52.679076 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:52.679025 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" podStartSLOduration=1.679008925 podStartE2EDuration="1.679008925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:52.677496179 +0000 UTC m=+2761.807232759" watchObservedRunningTime="2026-09-15 21:22:52.679008925 +0000 UTC m=+2761.808745507" Sep 15 21:22:53.044447 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:53.044394 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-2c6tb_1e360961-0f5b-42b6-8a9b-576067c2e536/volume-data-source-validator/0.log" Sep 15 21:22:54.083480 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:54.083447 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25vtb_8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b/dns/0.log" Sep 15 21:22:54.102996 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:54.102953 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25vtb_8ff12cc4-90d7-46ce-83d4-6ceb6a9dc77b/kube-rbac-proxy/0.log" Sep 15 21:22:54.211890 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:54.211859 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ssxn_0ecf3d66-c8a6-4ecb-b83d-ddb7e8db3809/dns-node-resolver/0.log" Sep 15 21:22:54.830192 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:54.830165 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vjcfq_a11c1990-58c3-4921-84c7-f7bf86deaddf/node-ca/0.log" Sep 15 21:22:55.757768 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:55.757735 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l_1364f2e9-1cb6-4810-9daf-99fd905f06ab/istio-proxy/0.log" Sep 15 21:22:56.250996 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:56.250961 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kwffw_2e53a04c-0eeb-4627-962b-a4fbb8dff2fe/istio-proxy/0.log" Sep 15 21:22:56.254656 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:56.254631 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kwffw_2e53a04c-0eeb-4627-962b-a4fbb8dff2fe/istio-proxy/1.log" Sep 15 21:22:56.289009 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:56.288973 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lqfgw_47e44d4f-614f-4962-8021-1b5dbc4edc74/payload-pre-processing/0.log" Sep 15 21:22:56.340689 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:56.340659 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f448b5b8-4bm9g_4bbf573e-e3c2-4d9d-8711-5a6cf974c302/router/0.log" Sep 15 21:22:57.264758 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:57.264721 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gnbcs_bd61f39f-8693-4da4-b81a-b84b3ade845f/serve-healthcheck-canary/0.log" Sep 15 21:22:57.775271 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:57.775225 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-27q29_9312d9bc-0d1b-48c8-8e90-ebe08397042f/insights-operator/0.log" Sep 15 21:22:57.776123 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:57.776091 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-27q29_9312d9bc-0d1b-48c8-8e90-ebe08397042f/insights-operator/1.log" Sep 15 21:22:57.855644 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:57.855614 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9b9vt_a87410fa-9f76-4091-bbad-2a10be1cc046/kube-rbac-proxy/0.log" Sep 15 21:22:57.872828 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:57.872801 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9b9vt_a87410fa-9f76-4091-bbad-2a10be1cc046/exporter/0.log" Sep 15 21:22:57.892059 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:57.892030 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9b9vt_a87410fa-9f76-4091-bbad-2a10be1cc046/extractor/0.log" Sep 15 21:22:59.684755 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:22:59.684721 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-nkstl" Sep 15 21:23:00.455560 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:00.455529 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8bg5g_f04b6918-b1ee-41ed-989f-a187aee1e26a/manager/0.log" Sep 15 21:23:00.524995 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:00.524965 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2p2gd_b8ef9dcd-5b1c-487b-a779-b551c321365c/server/0.log" Sep 15 21:23:00.649003 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:00.648970 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-w7hvt_d3a611af-d53e-409c-a463-63b98690a3b2/manager/0.log" Sep 15 21:23:00.738238 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:00.738158 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-4fgfr_69d81a70-c5a5-4b80-9b20-cee452b4e24c/manager/0.log" Sep 15 21:23:00.757463 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:00.757436 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-4fgfr_69d81a70-c5a5-4b80-9b20-cee452b4e24c/copy-manifests/0.log" Sep 15 21:23:01.961167 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:01.961073 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l_1364f2e9-1cb6-4810-9daf-99fd905f06ab/istio-proxy/0.log" Sep 15 21:23:02.219014 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:02.218942 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-675cf9bd68-fkl5q_85d18072-74e1-4ccd-a4ba-57078cace683/manager/0.log" Sep 15 21:23:02.423917 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:02.423886 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kwffw_2e53a04c-0eeb-4627-962b-a4fbb8dff2fe/istio-proxy/0.log" Sep 15 21:23:02.427214 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:02.427189 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kwffw_2e53a04c-0eeb-4627-962b-a4fbb8dff2fe/istio-proxy/1.log" Sep 15 21:23:02.451660 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:02.451617 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lqfgw_47e44d4f-614f-4962-8021-1b5dbc4edc74/payload-pre-processing/0.log" Sep 15 21:23:02.489370 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:02.489293 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f448b5b8-4bm9g_4bbf573e-e3c2-4d9d-8711-5a6cf974c302/router/0.log" Sep 15 21:23:03.993162 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:03.993128 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-t2p5l_1364f2e9-1cb6-4810-9daf-99fd905f06ab/istio-proxy/0.log" Sep 15 21:23:04.486961 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:04.486925 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kwffw_2e53a04c-0eeb-4627-962b-a4fbb8dff2fe/istio-proxy/0.log" Sep 15 21:23:04.490094 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:04.490069 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kwffw_2e53a04c-0eeb-4627-962b-a4fbb8dff2fe/istio-proxy/1.log" Sep 15 21:23:04.514975 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:04.514940 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lqfgw_47e44d4f-614f-4962-8021-1b5dbc4edc74/payload-pre-processing/0.log" Sep 15 21:23:04.551471 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:04.551434 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f448b5b8-4bm9g_4bbf573e-e3c2-4d9d-8711-5a6cf974c302/router/0.log" Sep 15 21:23:08.917922 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:08.917893 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4jtq_43127936-57a8-463b-ac1b-2568e6fa2347/kube-multus-additional-cni-plugins/0.log" Sep 15 21:23:08.940070 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:08.940047 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4jtq_43127936-57a8-463b-ac1b-2568e6fa2347/egress-router-binary-copy/0.log" Sep 15 21:23:08.960294 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:08.960276 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4jtq_43127936-57a8-463b-ac1b-2568e6fa2347/cni-plugins/0.log" Sep 15 21:23:08.978970 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:08.978941 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4jtq_43127936-57a8-463b-ac1b-2568e6fa2347/bond-cni-plugin/0.log" Sep 15 21:23:09.001053 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:09.001028 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4jtq_43127936-57a8-463b-ac1b-2568e6fa2347/routeoverride-cni/0.log" Sep 15 21:23:09.021965 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:09.021912 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4jtq_43127936-57a8-463b-ac1b-2568e6fa2347/whereabouts-cni-bincopy/0.log" Sep 15 21:23:09.045845 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:09.045815 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4jtq_43127936-57a8-463b-ac1b-2568e6fa2347/whereabouts-cni/0.log" Sep 15 21:23:09.256609 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:09.256583 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ckvww_b2d6afa5-55c1-484f-9539-b6d240fdf155/kube-multus/0.log" Sep 15 21:23:09.316755 ip-10-0-139-201 kubenswrapper[2580]: I0915 21:23:09.316661 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5vvbb_dad0237e-3438-4e30-a3f8-5f76d25d0763/network-metrics-daemon/0.log"